Querying and Routing Logs with Cloud Logging

Overview and What You Will Learn

In this lab, you will generate log entries from a VM, write increasingly specific query filters to narrow down to exactly the relevant entries, and configure a log sink routing specific logs to BigQuery for long-term, queryable retention.

Why This Matters in Production

An engineer is told "checkout is slow sometimes," with no more specific information than that. Without the ability to actually filter logs - by resource, by severity, by a specific time window - finding the actual root cause becomes a matter of scrolling through a huge volume of raw text manually. Cloud Logging's filter syntax turns that into a precise, repeatable question asked directly against the data.

Core Principles

Cloud Logging collects log entries from every GCP service and your own applications into a searchable, filterable store - and its query filter syntax is the tool that turns "something is wrong" into "here is the exact log entry explaining why."

Detailed Step-by-Step Practical Lab

1. Create a project and a VM that will generate some log activity:

2. Start with the simplest possible query - view recent logs for this specific VM:

3. Narrow the query to only error-severity entries from the last hour:

4. Combine multiple conditions - error-severity entries specifically for Compute Engine resources:

5. Create a BigQuery dataset to serve as a log sink destination for long-term retention:

6. Create a Log Sink routing error-severity logs to that BigQuery dataset:

7. Grant the sink's service account permission to write to the BigQuery dataset:

Note

Every Log Sink has its own dedicated service account (the writerIdentity) used specifically to write matching logs to the destination - this identity must be explicitly granted write permission on the destination, since the sink itself doesn't automatically have access just by existing.

8. Clean up:

Production Best Practices & Common Pitfalls

Common Mistake

Relying solely on Cloud Logging's default retention window for logs that genuinely need long-term retention for compliance or historical analysis. Cloud Logging's own storage has a default retention period - a Log Sink routing relevant logs to BigQuery or Cloud Storage is necessary for retention beyond that default window.

Tip

Build query filters incrementally - start broad, confirm results look reasonable, then add conditions one at a time (resource type, severity, specific field values) rather than trying to write the complete, precise filter in one attempt.

• Log Sinks only route logs matching their filter going forward from creation, not retroactively. Historical logs that existed before the sink was created are not backfilled into the new destination automatically.
• Audit Logs are a specific category within Cloud Logging worth filtering for separately during a security investigation - logName:"cloudaudit.googleapis.com" narrows a query specifically to administrative and data-access audit entries, distinct from regular application or system logs.

Quick Reference & Troubleshooting Commands