GCP Interview Questions

A Google Cloud project is the container that every resource (VMs, buckets, databases, service accounts, APIs) lives inside. Nothing in GCP exists outside of a project.

This matters because the project is GCP's real security and billing boundary, not the organization or a folder. IAM policies, enabled APIs, and quotas are all attached at the project level by default, so "which project is this resource in" is usually the first question to ask when something is unreachable or a bill looks wrong.

A project has three identifiers:

• Project ID: globally unique, cannot be changed, used in gcloud commands and URLs.
• Project number: assigned automatically by Google.
• Display name: editable and does not need to be unique.

Reference: Creating and managing projects

Compute Engine gives you raw virtual machines that you configure and manage yourself. App Engine is a managed platform where you deploy application code and Google handles servers, scaling, and patching.

Choose Compute Engine when you need control over the OS, custom drivers, a specific kernel, or special licensing. Choose App Engine when you have a standard web app or API in a supported language and do not want to think about infrastructure. It handles traffic splitting, versioning, and automatic scaling for you.

The trade-off is flexibility versus operational work. App Engine Standard has real limits (fixed runtimes, no arbitrary binaries, a sandboxed environment), so complex workloads usually move to Compute Engine, containers, or GKE.

Reference: Compute Engine documentation

A bucket is the top-level container you create in Cloud Storage to hold objects (files). Every object belongs to exactly one bucket, and bucket names are globally unique across all of GCP, not just your project.

The location type you choose at creation decides where the data lives:

• Region: one geographic area, lowest latency to nearby compute.
• Dual-region: two specific regions, good resilience with decent latency.
• Multi-region: a large area (such as the whole US), highest availability.

You cannot change a bucket's location later. Moving means creating a new bucket and copying the objects, so decide up front, usually by placing the bucket in the same region as the compute that reads from it most.

Reference: Bucket locations

BigQuery is Google Cloud's fully managed, serverless data warehouse. You run SQL over very large datasets without provisioning any servers.

Its key design choice is that storage and compute are separate. Your data sits in Google's distributed columnar storage, and each query borrows compute (called "slots") only while it runs. That is why it can scan terabytes in seconds, and why cost is usually driven by how much data a query scans rather than by hardware you rented.

Use it for analytical work (aggregating across huge tables). Do not use it as the main database of an application, where you need many small reads and writes with strict consistency.

Reference: BigQuery overview

A VPC (Virtual Private Cloud) in GCP is a private, software-defined network that gives your resources their own IP space, routing, and firewall rules.

The big difference from AWS is scope. A GCP VPC is a global resource, and its subnets are regional. One VPC can have subnets in many regions, and VMs in those subnets talk privately with no VPN or peering. In AWS a VPC lives in one region, so connecting regions needs peering or Transit Gateway.

So in GCP networking the useful question is usually "which region is this subnet in?" rather than "which VPC?".

Reference: VPC network overview

Owner, Editor, and Viewer are the original "basic roles". They grant broad permissions across almost every service in a project. Viewer can read everything, Editor can read and change everything, and Owner can also manage IAM and billing.

They come up in interviews because they are a known anti-pattern. Giving Editor to a person or service account hands over write access to dozens of services it will never use. That breaks least privilege and makes audits meaningless.

The expected answer: production access should use predefined roles (curated by Google for one service, such as roles/bigquery.dataViewer) or custom roles (a hand-picked permission list). Keep basic roles for personal sandboxes or real full-project ownership.

Reference: Understanding roles

GKE is Google's managed Kubernetes service. Google runs and patches the control plane (API server, etcd, scheduler). You manage the worker nodes in Standard mode, or nothing at the node level in Autopilot mode.

It exists because running Kubernetes yourself means operating a distributed system on top of a distributed system: upgrading the control plane, securing etcd, and recovering from node failures are all hard. GKE removes the control plane work, and Autopilot also removes node management, billing per pod resource request instead of per VM.

It is the natural choice when a team already thinks in Kubernetes terms (Deployments, Services, CRDs).

Reference: GKE overview

Cloud Run is Google's serverless platform for running containers. You give it a container image, and it handles scaling, load balancing, and servers. It can scale all the way down to zero instances when there is no traffic.

Unlike Cloud Functions, which deploys one function from source code, Cloud Run deploys a whole container. That means any language, any framework, and many routes in one service.

On the "how much do I manage" scale it sits between Cloud Functions (simplest) and GKE (most control): you get container flexibility without running a cluster.

Reference: Cloud Run overview

A service account is an identity for a workload instead of a person. An application, a VM, or a CI/CD pipeline signs in as a service account to call GCP APIs, rather than using someone's password.

It is both an identity (you can grant IAM roles to it) and a resource (you can control who is allowed to impersonate it). It has an email-style name and can be attached to compute resources or used to create short-lived credentials.

Best practice: give each workload its own service account with only the permissions it needs, and avoid long-lived key files.

Reference: Service accounts overview

A region is a geographic area such as us-central1. A zone is an isolated deployment area inside a region, such as us-central1-a, with its own power, cooling, and networking.

Every region has at least three zones. Zones in a region are connected by fast links but are built to fail independently, so a power problem in one zone should not take down another.

This is why high availability usually means spreading across multiple zones in a region, and disaster recovery means using more than one region.

Reference: Regions and zones

Cloud SQL is GCP's managed relational database service for MySQL, PostgreSQL, and SQL Server. Google handles patching, backups, replication, and failover.

Pick it when an application needs a normal relational database and you do not want to run database servers yourself. It supports read replicas for scaling reads, and a high-availability mode that fails over to a standby in another zone.

What it does not do is scale writes horizontally. Writes go to one primary, so beyond a single instance's limits you look at Cloud Spanner or a sharding design.

Reference: Cloud SQL overview

Cloud CDN caches content at Google's edge locations near users. A load balancer spreads incoming requests across your backends. They solve different problems and are normally used together.

The load balancer's job is routing and health checks: choosing which healthy backend serves a request. Cloud CDN's job is to avoid the backend completely for content that has not changed, by serving a cached copy from a nearby edge.

In GCP, Cloud CDN is a feature you turn on for a global external Application Load Balancer backend, not a separate standalone product.

CDN helps with static or slow-changing content (images, JavaScript, cacheable API responses). The load balancer still handles every request, including all dynamic traffic.

Reference: Cloud CDN overview

Cloud Storage is object storage reached over HTTP APIs, for whole files with no filesystem. Persistent Disk is block storage that attaches to a VM or GKE node and works like a normal disk the OS can format and mount.

The difference that matters is access pattern. Cloud Storage objects are replaced whole, not edited in place. They can be reached from anywhere with the right credentials, scale almost without limit, and have very high durability. A Persistent Disk allows writes at any byte position and a filesystem on top, but it is tied to a VM (or two zones for regional disks).

A common mistake is putting uploads, reports, or backups on a Persistent Disk out of habit. Cloud Storage is both cheaper and the right fit for that.

Reference: Persistent Disk overview

Pub/Sub is Google Cloud's managed messaging service. A sender (publisher) puts a message on a topic, and one or more receivers (subscribers) get it through a subscription.

It solves the problem of tightly linked services. The sender does not need to know who receives the message, how many receivers there are, or whether they are online right now. Messages wait in the subscription until they are processed.

Typical uses: sending events between microservices, absorbing traffic spikes, and feeding data into pipelines such as Dataflow and BigQuery.

Reference: Pub/Sub overview

Cloud Bigtable is a fully managed, wide-column NoSQL database built for very large amounts of data with very low latency. It is the same kind of technology Google uses internally for services like Search and Maps.

It is good for huge, fast-growing data that is read and written by a key: time-series data, IoT sensor readings, ad-tech events, and user activity logs. It handles millions of operations per second.

It is not good for complex SQL queries, joins, or multi-row transactions. You look up data by row key, so designing that key well is the most important skill with Bigtable.

Reference: Bigtable overview

Firestore is a serverless NoSQL document database. You store data as documents (like JSON objects) inside collections, instead of rows inside tables.

The differences from Cloud SQL:

• Schema: Firestore documents can have different fields. Cloud SQL tables have a fixed schema.
• Queries: Firestore does not support joins. You shape your data around how you read it.
• Scaling: Firestore scales automatically and has built-in real-time listeners and offline support, which is why it is popular for mobile and web apps.

Choose Firestore for flexible, app-centric data that needs easy scaling. Choose Cloud SQL when you need joins, strict relational structure, or existing SQL tooling.

Reference: Firestore overview

Secret Manager is a GCP service that stores sensitive values such as API keys, passwords, and certificates. It keeps them encrypted, controls access with IAM, and records who read them.

Putting secrets in code, config files, or plain environment variables is risky because they end up in Git history, container images, and logs, and they are hard to rotate. Secret Manager gives you:

• Versions: add a new version to rotate a secret without changing the name.
• IAM control: only the service accounts you allow can read it.
• Audit logs: you can see every access.

Cloud Run and GKE can read secrets directly, so the app gets the value at runtime and it never sits in your source code.

Reference: Secret Manager overview

Cloud DNS is GCP's managed, highly available Domain Name System service. It turns names such as app.example.com into IP addresses, and it runs on Google's global network.

You can create public zones (answer queries from the internet) and private zones (answer only inside your VPC, useful for internal names like db.internal).

You pay for zones and queries, and you do not manage any DNS servers. It also supports DNSSEC for signing records.

Reference: Cloud DNS overview

Memorystore is GCP's managed in-memory data store. It offers Redis, Valkey, and Memcached, and Google handles patching, replication, and failover.

Use it as a fast cache in front of a slower database, for session storage, rate limiting, leaderboards, and short-lived data. Reads come from memory in under a millisecond, which lowers load and cost on the main database.

Remember that a cache is a copy, not the source of truth. Design the app so it still works (just slower) if the cache is empty or unavailable.

Reference: Memorystore overview

Artifact Registry is GCP's managed place to store and manage build artifacts: container images, plus packages such as npm, Maven, Python, and Go modules. It replaced the older Container Registry.

It is where a CI/CD pipeline pushes an image, and where GKE or Cloud Run pulls it from. It uses IAM for access, supports vulnerability scanning, and can be regional, which keeps images close to the services that run them.

Reference: Artifact Registry overview

Cloud Audit Logs record who did what, where, and when in your GCP projects. They are the main tool for security reviews and for answering "what changed?".

The main types:

• Admin Activity: changes to configuration or metadata (for example creating a VM or changing IAM). Always on, free, and cannot be turned off.
• Data Access: reads and writes of user data. Off by default for most services because it can be very large (BigQuery is an exception), so you turn it on where needed.
• System Event: changes made by Google systems, such as live migration.
• Policy Denied: records when access was blocked by a security policy.

Reference: Cloud Audit Logs overview

gcloud is the command-line tool for working with GCP. You use it to create resources, deploy apps, and manage IAM from a terminal or a script.

Settings live in a configuration. The key ones are the active account and project:

A common mistake is running a command against the wrong project because the default was never checked. You can keep several named configurations (gcloud config configurations create) for dev and prod and switch between them.

Reference: gcloud CLI overview

Cloud Armor is GCP's web application firewall (WAF) and DDoS protection service. It sits in front of an external Application Load Balancer and filters traffic before it reaches your backends.

You write security policies with rules, for example: block certain countries or IP ranges, rate limit a noisy client, or block common attacks such as SQL injection and cross-site scripting using preconfigured rules.

It works at the edge of Google's network, so attack traffic is stopped before it uses your capacity.

Reference: Cloud Armor overview

Dataflow is GCP's fully managed service for running data processing pipelines, for both streaming (continuous) and batch (one-time) data. It runs pipelines written with the Apache Beam SDK.

You describe what to do with the data (read, transform, aggregate, write), and Dataflow handles the workers, scaling, and retries. A classic pipeline reads from Pub/Sub, transforms events, and writes to BigQuery.

The main benefit is that one programming model covers both batch and streaming, and you do not manage a cluster.

Reference: Dataflow overview

Dataproc is GCP's managed service for running Apache Hadoop and Spark (and related tools such as Hive and Flink). You create a cluster in about a minute, run your job, and delete the cluster.

It is mostly used to move existing Spark or Hadoop jobs to the cloud with few changes. Because clusters start quickly and are billed per second, a good pattern is ephemeral clusters: create one for a job, then delete it, instead of leaving it running.

Dataproc also has a serverless option, where you submit a Spark job and Google handles the cluster.

Reference: Dataproc overview

Choose Cloud Functions for small, single-purpose code that reacts to an event (a webhook, an image resize, a Pub/Sub message handler). Choose Cloud Run when you have a full containerized app or API with several routes and custom dependencies.

The mechanical difference is what you deploy. With Cloud Functions you give Google source code and a dependency file, and it builds the container for you, with one entry point. With Cloud Run you give a container image you built, which can run any language and expose any number of routes.

Both scale to zero and both can handle HTTP or events. In practice the deciding factor is the team: if you already use Docker, Cloud Run fits your workflow. If you want to skip containers for a small piece of glue code, Cloud Functions removes that step.

Reference: Choosing a serverless option

It comes down to how much infrastructure control you need versus how much you want Google to manage. The five options sit on a scale from "you manage everything" to "you manage nothing".

The common wrong answer is picking GKE by default because it sounds the most serious. Most new stateless services should start on Cloud Run. Teams move to GKE only when they hit a concrete Kubernetes need (StatefulSets, DaemonSets, a service mesh).

Reference: Choosing a compute option

Predefined roles are made by Google for one service, such as roles/pubsub.publisher. They bundle exactly the permissions that job needs, and Google updates them as the service changes. Custom roles are ones you build from a list of individual permissions. You get exact control, but you must maintain them.

Start with a predefined role because it is maintained for you. Write a custom role when the closest predefined one is either too broad (it includes a permission you do not want to grant) or too narrow (it lacks one permission you need, and the next role up grants ten more).

The cost of a custom role is upkeep: as the service adds new permissions, you must review and update the role yourself.

Reference: Creating and managing custom roles

The hierarchy is Organization, then Folders (optional, can be nested), then Projects, then individual resources. It exists so you can set access and policy once near the top and have it apply to everything below.

An IAM policy on the organization applies to every folder and project under it. A policy on a folder applies to every project in it. A policy on a project applies to every resource in it.

The key detail: inheritance only adds. A lower level can grant more access, but it cannot take away a permission granted higher up. If someone is Viewer at the org level, you cannot remove that at the project level.

Many people assume IAM works like a firewall with allow and deny at each layer. It does not. To reduce access you narrow the grant where it was made, or restructure the hierarchy. Organization Policies (a separate feature) can add restrictions, but IAM role bindings only add up.

Reference: Resource hierarchy

Shared VPC lets many projects use one VPC owned by a host project. Resources in the service projects are placed in subnets they do not own. VPC Peering connects two separate, independently owned VPCs so they can talk over internal IPs while staying separate networks.

Use Shared VPC when a central network team wants to manage IP ranges, firewall rules, and routes once for many teams. All projects must be in the same organization.

Use VPC Peering to connect networks managed by different owners, even across organizations (for example, to a SaaS vendor's VPC, or to a company you just acquired).

Know this trade-off: peering is not transitive. If A peers with B and B peers with C, A cannot reach C through B. Shared VPC is one flat network, so it does not have that limit.

Reference: Shared VPC overview

GCP load balancers vary along two choices: global or regional, and external or internal. You also choose the layer (HTTP at Layer 7, or TCP/UDP at Layer 4).

• Global external Application Load Balancer: Layer 7, one anycast IP, sends users to the closest healthy backend worldwide. Use for public web apps, URL-based routing, and Cloud CDN.
• Regional external Application or Network Load Balancer: same idea inside one region, useful when rules say traffic must stay in a region.
• Internal Application or Network Load Balancer: private IP only, for service-to-service traffic inside a VPC.
• Passthrough Network Load Balancer: Layer 4, keeps the client's real source IP, good for non-HTTP protocols.

The interviewer is checking that you know it is not "one load balancer". You choose by protocol, scope (does it span regions?), and whether traffic is public or internal.

Reference: Choosing a load balancer

BigQuery is a serverless analytical warehouse for scanning huge datasets with SQL. Cloud SQL is a managed transactional database for an application's everyday reads and writes.

The difference is the shape of the work. BigQuery stores data by column and runs queries in massive parallel, which is great for "total this column across a billion rows" but poor for "update one row and read it back". Many tiny queries are also expensive there. Cloud SQL stores data by row and supports transactions, indexes, and foreign keys like a normal application database, but it cannot match BigQuery's analytical throughput.

A common pattern: Cloud SQL is the system of record for the app, and the data is streamed or exported into BigQuery (often with Datastream or Dataflow) so analysts do not slow down production.

Reference: BigQuery overview

Object Versioning keeps older versions of an object when it is overwritten or deleted, so a change is not destructive. Lifecycle management lets you set rules that move or delete objects automatically based on age or other conditions.

With versioning on, deleting an object only marks the live version as "noncurrent". The data stays, so you can recover from an accidental delete. But every version is billed as its own object, so versioning left on forever quietly grows your bill.

Lifecycle rules fix that. For example: move objects to Nearline after 30 days, Coldline after 90, and delete noncurrent versions after 365. Teams often enable versioning and forget the second half.

Reference: Object Versioning

Cloud Storage has four classes: Standard, Nearline, Coldline, and Archive. Colder classes have a lower storage price but a higher retrieval cost and a longer minimum storage time.

• Standard: data used often (active sites, data being processed).
• Nearline: read about once a month or less (monthly backups).
• Coldline: read about once a quarter or less (disaster recovery copies).
• Archive: almost never read (long-term compliance data).

All four have the same latency and durability. Only price and minimum duration differ.

The common mistake is choosing by how important the data is instead of how often it is read. Critical data you never read can sit in Archive. Unimportant data you read all day belongs in Standard. Lifecycle rules can move data to colder classes automatically as it ages.

Reference: Storage classes

Workload Identity lets a pod call Google Cloud APIs as a Google service account (GSA) without a downloaded key file. It links a Kubernetes service account (KSA) to a GSA, and the pod's requests are exchanged for short-lived credentials.

The alternative is mounting a JSON key as a file or a Kubernetes secret. That key is long-lived, is never rotated automatically, and can leak through logs, container images, or a compromised pod.

Workload Identity credentials are short-lived, tied to one KSA-to-GSA link, and never sit on disk as a static file. That removes a whole class of key-leak incidents. The weak answer is "just mount the key, it's simpler". The strong answer names the actual risk (long-lived, unrotated, stealable credentials) and how this closes it.

Reference: Workload Identity

Spot VMs are spare Compute Engine capacity sold at a big discount (roughly 60 to 91 percent off on-demand). The catch is that Google can take them back with very short notice when it needs the capacity.

They suit workloads that are stateless, fault tolerant, and cheap to restart: batch jobs, CI/CD runners, rendering, and the non-critical, replicated parts of a GKE cluster. They are a bad fit for anything that cannot survive a sudden stop, such as a single-replica database or a long job with no checkpoints.

In GKE the usual pattern is a separate Spot node pool with a taint. Only workloads with a matching toleration (ones you decided can handle preemption) are scheduled there, so a stateful workload never lands on temporary capacity by accident.

Reference: Spot VMs

Ingress rules control traffic coming into a resource. Egress rules control traffic going out. By default every VPC has an implied rule that allows all egress and denies all ingress from outside.

Default-deny ingress is on purpose: nothing is reachable from outside until you write an allow rule. Rules are checked by priority (lower number wins). You can target a rule at specific VMs by network tag or service account, which is the best way to keep access tight (for example, allow port 443 only for VMs tagged web-server).

A common trap: people think outbound-only workloads need an explicit egress allow rule. They do not, because egress is open by default. You only add an egress deny rule when you want to restrict outbound traffic.

Reference: VPC firewall rules overview

IAM Conditions let you attach a rule to a role binding so that access is granted only when the condition is true. Common conditions use time, resource name or type, or request attributes.

Roles answer "what can this identity do". They have no idea of "only until Friday" or "only for resources whose name starts with team-a-". Conditions fill that gap.

Examples:

• Temporary access that expires by itself: request.time < timestamp("2026-12-31T00:00:00Z")
• A role that only applies to objects with a certain prefix, not the whole bucket.

In an interview this shows you know IAM is more than "assign a role". Conditions avoid creating a narrow custom role just to express a temporary or scoped grant.

Reference: IAM Conditions overview

In Standard, you choose and manage node pools: machine types, node counts, upgrades, and hardening. In Autopilot, Google manages the nodes. You only describe what your pods need, and you are billed for the CPU, memory, and storage your pods request, not for whole VMs.

Autopilot applies secure defaults automatically (Shielded Nodes, Workload Identity, no privileged pods) and removes tasks like patching and right-sizing nodes.

The trade-off is control. Autopilot limits some node-level features: DaemonSets have restrictions, some privileged operations are blocked, and you cannot SSH into nodes.

A practical rule: start with Autopilot for normal stateless workloads, and move to Standard only when you have a concrete need Autopilot cannot meet, such as special hardware or node agents that need broad privileges.

Reference: GKE Autopilot overview

Cloud Build is GCP's managed CI/CD service. It runs a list of build steps, each one a container, defined in a YAML file, usually triggered by a push to a repository.

A typical pipeline: a push to a connected repo triggers Cloud Build, which runs tests, builds a container image, pushes it to Artifact Registry, then deploys to Cloud Run, GKE, or App Engine. Production deploys are often gated by a manual approval or a canary rollout.

Security-minded pipelines add Binary Authorization at deploy time, so only images that passed the pipeline's checks (and were signed) are allowed to run.

Reference: Cloud Build overview

Partitioning splits a table into segments by a column (usually a date). Clustering sorts the data inside those segments by other columns. Both let a query skip data it does not need, which lowers the bytes billed.

On-demand BigQuery charges by bytes scanned, not rows returned. A query with WHERE date = '2026-09-01' on an unpartitioned table still scans the whole table. If the table is partitioned by that date, it reads only one day.

Clustering goes further. If you often filter by customer_id, clustering by it lets BigQuery skip blocks of non-matching rows even inside one day.

Nuances worth knowing: a table can have at most 4,000 partitions, and partitioning works best on bounded columns like dates. Clustering handles higher-cardinality columns, and the two can be combined.

Reference: Partitioned tables

A MIG autoscaler watches a signal you choose (CPU use, a Cloud Monitoring metric, load balancer capacity, or a schedule) and adds or removes VMs to keep that signal near your target.

You set a minimum and maximum instance count as guardrails. The autoscaler works inside that range.

Interviewers often ask about the cooldown period. A new VM is left out of the average for a set time (default 60 seconds) so the autoscaler does not misread its low early load and scale back down too soon.

MIGs also have autohealing, which is separate. If a health check fails on a VM, the MIG recreates it. That is about replacing broken instances, not about load.

Reference: Autoscaling groups of instances

Cloud SQL is a managed MySQL, PostgreSQL, or SQL Server instance. It scales up and out with read replicas. Cloud Spanner is Google's globally distributed relational database that scales writes and reads horizontally while keeping strong consistency and SQL.

The dividing line is write volume and geographic spread. Cloud SQL writes go through one primary, limited by that machine. Spanner spreads data over many nodes and regions and still keeps transactions consistent, which is very hard to build yourself.

The price: Spanner costs more and has a steeper design curve (choosing primary keys to avoid hotspots, interleaved tables). It is overkill for an app that will never outgrow one Cloud SQL instance.

The honest answer is "use Cloud SQL until measurements show you need Spanner". Choosing Spanner by default is overengineering.

Reference: Cloud Spanner overview

Private Google Access lets VMs that have only internal IP addresses reach Google APIs (such as Cloud Storage or BigQuery) without an external IP and without going over the public internet.

You need it when you removed external IPs from VMs for security but they still must call Google APIs, for example to read a bucket or write logs. Without it, a VM with no external IP cannot reach *.googleapis.com at all.

It is switched on per subnet, not per VM. Do not confuse it with Private Service Connect, which privately reaches specific published services (including third-party ones) rather than Google APIs in general.

Reference: Private Google Access

In synchronous processing the caller waits until the work is done and gets the result back, like a normal HTTP API call. In asynchronous processing with Pub/Sub the caller publishes a message and moves on, and one or more subscribers handle it later at their own pace.

Pub/Sub decouples sender and receiver. The publisher does not need to know who is listening or whether they are up right now. A slow subscriber does not slow the publisher.

Use it when work can wait, when one event must go to several independent consumers (one updates a search index, another sends a notification), or when you must absorb traffic bursts, since messages queue up instead of overwhelming a service.

The trade-off: you lose the immediate answer, and delivery is "at least once", so a subscriber may get the same message twice. Processing must be idempotent (safe to repeat).

Reference: Pub/Sub overview

GCP's observability tools work together: Cloud Logging collects logs, Cloud Monitoring tracks metrics and sends alerts, and Cloud Trace shows request latency across services. Cloud Run, GKE, and App Engine send data to all three with almost no setup.

In practice, build alerting policies on the signals that show real problems (error rate, latency percentiles, saturation) instead of reading raw logs. Use Trace when a request crosses several services, since it shows where the time is actually spent.

To centralize logs or use outside tools, create sinks that export logs to BigQuery, Cloud Storage, Pub/Sub, or third-party tools such as Grafana or Splunk.

Reference: Cloud Monitoring overview

Binary Authorization is a deploy-time check for GKE and Cloud Run. It lets only container images that are signed by an authority you trust run in your environment. The signature is usually proof the image passed your build, tests, or a vulnerability scan.

It protects against someone deploying an image that skipped your normal process: built on a laptop, never scanned, never tested. This can happen through a stolen credential, a misconfigured pipeline, or an engineer under pressure.

Without it, IAM permission to deploy is the only gate, and IAM says nothing about what is inside the image. With a policy, even a person with deploy rights cannot run an unsigned image, because the platform rejects it at admission time.

It works best when your CI pipeline signs images automatically after tests and scans pass.

Reference: Binary Authorization overview

An external load balancer has a public IP and accepts traffic from the internet. An internal one has a private IP that is reachable only inside your VPC (or connected networks) and is for service-to-service traffic.

"Internal or external" and "Layer 4 or Layer 7" are two separate choices. An internal Application Load Balancer is a common way to route HTTP traffic between microservices without exposing anything publicly.

This matters because of a real mistake: creating an external load balancer in front of an internal-only service (an admin API, a database proxy) exposes it to the internet. Know which type you are creating, and do not rely on a firewall rule alone to catch it.

Reference: Load balancing overview

Least privilege means giving an identity only the permissions it needs, so a stolen or misused credential can do as little damage as possible. IAM Recommender finds where this has drifted by checking which permissions an identity actually used in the last 90 days and suggesting a smaller role.

Roles pile up over time. Someone gets Editor "to unblock a task" and keeps it. A service account gets a broad role early and uses a fraction of it. Recommender shows which granted permissions were never used and proposes a narrower or custom role that would have covered the real activity.

It is a recommendation, not enforcement. Accepting it blindly can break a workload that uses a permission rarely (a monthly batch job, for example). Review each suggestion against known rare usage before applying.

Reference: IAM Recommender overview

Cloud NAT lets VMs with no external IP start outbound connections to the internet (to download packages, call an external API) without allowing any inbound connections from the internet.

A VM with only an internal IP is unreachable from outside, which is the security goal. But "cannot be reached" is different from "cannot reach out", and VMs often still need to patch the OS or call a third-party service.

Cloud NAT translates the VM's internal IP to a shared external IP only for outbound traffic and never accepts new inbound connections. So the "no external IP" protection stays, while egress still works.

Do not confuse it with Private Google Access, which covers reaching Google APIs privately. Cloud NAT is the general answer for reaching any external destination.

Reference: Cloud NAT overview

An instance template is a saved recipe for a VM: machine type, boot image, disks, network, service account, labels, and startup script. A Managed Instance Group (MIG) uses a template to create identical VMs.

The template is immutable. You cannot edit it. To change a setting, you create a new template and tell the MIG to use it, then run a rolling update that replaces VMs gradually.

This is why MIGs are good for stateless services: any VM can be deleted and rebuilt from the template at any time. You control the rollout with settings such as maxSurge (extra VMs created during the update) and maxUnavailable (how many may be down at once).

Reference: Instance templates

They all capture VM data, but at different scopes.

Snapshots are incremental, so after the first one only changed blocks are stored, which keeps cost low. A snapshot schedule gives you automatic backups.

A common trap: a snapshot does not include VM settings (machine type, network, metadata). If you need to rebuild the full VM, use a machine image.

Reference: Disk snapshots

Live migration moves a running VM to another physical host during Google's maintenance (hardware or software updates) without rebooting it. Your VM keeps running, and you usually see only a brief pause.

It does not apply to every VM. Spot VMs cannot live migrate and are stopped instead. VMs with GPUs, and some special hardware, typically cannot live migrate either, so for those you choose a maintenance policy of terminate (and set automatic restart).

The practical point: for most general VMs you get maintenance with no downtime, but for GPU and Spot workloads you must design for a restart.

Reference: Live migration

All three lower Compute Engine cost, but in different ways.

• Sustained use discounts: automatic discounts when a VM runs for a large part of the month. No commitment, but they apply only to some machine families.
• Committed use discounts (CUDs): you commit to a level of resources for 1 or 3 years and get a bigger discount. Good for a steady baseline you know you will keep.
• Spot VMs: the deepest discount, but Google can stop the VM at any time. Good for fault-tolerant, restartable work.

A sensible mix: cover your steady baseline with CUDs, run flexible or batch work on Spot, and let everything else be on-demand. The mistake to avoid is committing to more than your real minimum usage, because you pay for the commitment whether or not you use it.

Reference: Committed use discounts

Both connect your on-premises network to GCP, but over different paths.

• Cloud VPN: an encrypted tunnel over the public internet. Fast to set up and cheap. Bandwidth is limited per tunnel, and latency can vary because it uses the internet. HA VPN gives a 99.99% SLA when configured with two tunnels.
• Cloud Interconnect: a private physical connection to Google, either dedicated (you connect at a Google location) or through a partner. It gives higher bandwidth, lower and steadier latency, and does not cross the public internet.

Choose VPN for getting started, smaller loads, or as a backup. Choose Interconnect for large or latency-sensitive traffic such as big data transfers or hybrid databases. Note that Interconnect is not encrypted by default, so add MACsec or VPN over it if encryption is required.

Reference: Choosing a network connectivity product

After Pub/Sub delivers a message, the subscriber has an ack deadline (10 seconds by default, up to 10 minutes) to acknowledge it. If it does not ack in time, or sends a negative ack, Pub/Sub treats it as not processed and redelivers it.

This gives "at least once" delivery. It also means a message that always fails (a "poison message") would be retried forever and could block progress.

The fix is a dead-letter topic. You set a maximum number of delivery attempts, and after that Pub/Sub moves the message to the dead-letter topic where you can inspect it later. You should also use a retry policy with exponential backoff so a failing message is not retried in a tight loop.

Because of redeliveries, your processing must be idempotent.

Reference: Dead-letter topics

In a pull subscription, your subscriber asks Pub/Sub for messages when it is ready. In a push subscription, Pub/Sub sends each message as an HTTPS request to an endpoint you provide.

Choose pull when you want control over speed and throughput. Choose push when the receiver is a serverless HTTP service and you want Pub/Sub to call it. For push, a 2xx response means success, and anything else triggers a retry.

Reference: Subscription types

Choose Dataflow for new pipelines, especially streaming, when you want no cluster to manage. It scales automatically and the Beam model handles both batch and streaming.

Choose Dataproc when you already have Spark or Hadoop code and want to move it with few changes, when you need specific open-source tools, or when your team already knows Spark well.

The trap answer is "Dataflow is always better because it is serverless". Rewriting a working Spark codebase in Beam costs real time. Dataproc is often the right call for a migration.

Reference: Dataflow vs Dataproc

In Bigtable, data is sorted by row key, and the only fast lookups are by key or key range. So the key decides both what you can query efficiently and how load is spread across nodes.

Two rules:

1. Design the key for your main read pattern. For time-series data, a key such as sensor_id#reversed_timestamp lets you read one sensor's latest data with a fast range scan.
2. Avoid hotspots. If the key starts with something that increases (like a timestamp), all new writes land on the same node while others sit idle. Put a well-spread value first (such as a device id), or use salting or hashing.

Changing a bad key later means rewriting the whole table, so this is a decision to get right up front. Also avoid very long keys and sensitive data in keys.

Reference: Schema design best practices

Start from the shape of the data and the access pattern.

Ask three questions: Do I need joins and transactions? How big and fast will it grow? Is the main use transactional or analytical?

The common mistake is choosing the most powerful option (Spanner or Bigtable) for a small app. Cloud SQL or Firestore are simpler and cheaper until measurements show otherwise.

Reference: Database options

All data in GCP is encrypted at rest by default with keys Google manages. With CMEK, you create and control the key yourself in Cloud KMS, and the service (Cloud Storage, BigQuery, Cloud SQL, and others) uses your key to encrypt the data.

Why use CMEK:

• Control: you can rotate, disable, or destroy the key. Disabling the key makes the data unreadable.
• Compliance: some rules require customer control of keys.
• Audit: key use is logged.

The trade-off is responsibility. If you delete or lose access to the key, the data is gone. You also pay for KMS and must manage permissions, since the service's account needs access to the key. Use CMEK when you have a real compliance or control need, not by default.

Reference: Customer-managed encryption keys

Give the workload's own service account the role roles/secretmanager.secretAccessor on only the secrets it needs, and let the platform inject the secret at runtime.

Good practice:

• Cloud Run can mount a secret as an environment variable or a file. GKE can use the Secret Manager add-on or the CSI driver.
• Reference a specific version in production for stability, or latest when you want automatic rotation to take effect on restart.
• Keep a separate service account for each workload so access is narrow.
• Turn on audit logs so you can see who read a secret.

Avoid reading all secrets at startup "just in case", granting access at the project level, or printing secret values in logs. For rotation, add a new version, update consumers, then disable the old version.

Reference: Using secrets in Cloud Run

Workload Identity Federation lets workloads outside GCP (GitHub Actions, AWS, Azure, an on-prem system) access GCP resources without a downloaded service account key. They prove who they are with their own identity token, and GCP exchanges it for short-lived credentials.

The problem it solves is the key file. A downloaded service account key is long-lived, can be copied, and is hard to rotate. With federation:

• No secret is stored in the external system.
• Credentials expire quickly.
• You can restrict access to a specific repository or branch with attribute conditions.

A common example: a GitHub Actions workflow deploys to Cloud Run using a token from GitHub, with no GOOGLE_CREDENTIALS secret at all. Interviewers like this answer because it shows you know how to remove long-lived keys.

Reference: Workload Identity Federation

Impersonation lets an identity (a person or another service account) temporarily act as a service account, without ever holding its key. The caller needs the role Service Account Token Creator on the target account and asks GCP for short-lived credentials.

Why it is safer:

• No key to leak or share. Credentials last minutes or hours.
• Clear audit trail. Logs show who impersonated which account.
• Easy to revoke. Remove the role and access stops.

A practical use: a developer runs gcloud ... --impersonate-service-account=deployer@project.iam.gserviceaccount.com to test with the pipeline's permissions, instead of getting a copy of its key. Note that Token Creator is powerful, because whoever has it effectively has the service account's permissions, so grant it carefully.

Reference: Service account impersonation

A signed URL is a link that gives time-limited access to one specific object, without the person needing a Google account or IAM permission. The link carries a signature made with a service account's credentials, and it stops working after the expiry time.

Typical uses:

• Let a user download a private file (an invoice) for 15 minutes.
• Let a browser upload straight to a bucket without sending the file through your server, which saves bandwidth and cost.

Key points: anyone who has the link can use it until it expires, so keep expiry short and send it over HTTPS. The access is limited by what the signing account can do. For many objects or long-term public content, consider other options such as IAM or Cloud CDN with signed cookies.

Reference: Signed URLs

Uniform bucket-level access turns off per-object ACLs and makes IAM the only way to control access to a bucket and everything in it.

Without it, access can come from two systems at once: IAM and object ACLs. That is confusing. Someone can see only IAM and think a bucket is private while a single object has an ACL that makes it public.

With uniform access:

• One place to look and audit.
• Easier to enforce rules with Organization Policy and IAM Conditions.
• Less chance of accidental public objects.

It is the recommended default for new buckets. Before enabling it on an old bucket, check whether any app depends on object ACLs, since those stop working. You can also add public access prevention to block public access entirely.

Reference: Uniform bucket-level access

Every time you deploy a change to a Cloud Run service, it creates a new, immutable revision. Old revisions stay available. The service sends traffic to one or more revisions according to the percentages you set.

This gives you safe rollouts:

• Canary: send 5% of traffic to the new revision, watch errors, then raise it.
• Instant rollback: point 100% back at the previous revision. No rebuild needed.
• Tags: give a revision its own URL to test it before it gets any live traffic.

A common mistake is deploying with "send 100% to latest" for a risky change. For important services, deploy with no traffic first, test via a tagged URL, then shift traffic gradually.

Reference: Rollbacks, gradual rollouts, and migration

They scale different things.

• HPA (Horizontal Pod Autoscaler): changes the number of pods based on metrics such as CPU or custom metrics.
• VPA (Vertical Pod Autoscaler): changes the CPU and memory requests of each pod to fit real usage.
• Cluster Autoscaler: changes the number of nodes. It adds nodes when pods are Pending because there is no room, and removes under-used nodes.

They chain together: load rises, HPA adds pods, pods do not fit, the Cluster Autoscaler adds nodes. When load falls, the steps reverse.

Two cautions. First, do not let HPA and VPA both act on CPU for the same workload, since they fight each other. Second, the Cluster Autoscaler reacts to pod requests, not real usage, so wrong requests give wrong scaling. In Autopilot, node scaling is handled for you.

Reference: GKE autoscaling

Release channels (Rapid, Regular, Stable, and Extended) let you choose how quickly your cluster receives new Kubernetes versions. Google upgrades the control plane and nodes automatically within the channel.

• Rapid: newest features first, more change risk.
• Regular: a balance, and the common default.
• Stable: versions that have been proven longer, for cautious teams.

Good upgrade habits:

• Use maintenance windows and exclusions so upgrades avoid busy periods.
• Set PodDisruptionBudgets so upgrades do not take down too many replicas at once.
• Test upgrades first in a staging cluster on the same channel.
• Check for removed Kubernetes APIs before a version jump.

Not upgrading is not an option: old versions leave support and stop receiving security fixes.

Reference: Release channels

In a private cluster, the nodes have only internal IP addresses, so they are not reachable from the internet. You can also make the control plane endpoint private, or limit it with authorized networks so only certain IP ranges can call the Kubernetes API.

Because nodes have no external IP, they cannot reach out by default. You provide outbound access with Cloud NAT. To pull images or call Google APIs, enable Private Google Access on the subnet, or use Artifact Registry in the same region.

Why use it: it shrinks the attack surface, since a node cannot be scanned or attacked directly from outside.

A frequent trap: pods fail to pull a public image and nobody notices that there is no Cloud NAT configured.

Reference: Private clusters

Both expose services in a cluster to outside traffic through a Google Cloud load balancer.

Ingress is the older Kubernetes object. It is simple, but limited: one resource mixes routing rules and infrastructure settings, and many features need vendor-specific annotations.

Gateway API is the newer, more expressive model. It splits roles:

• GatewayClass and Gateway: managed by the platform team (the load balancer itself).
• HTTPRoute: managed by app teams (routing rules for their service).

It supports richer features such as header-based routing, traffic weights for canary releases, and multi-cluster setups, using standard fields instead of annotations.

For new work, Google recommends Gateway API. Keep Ingress for existing, simple setups that work and do not need the extra features.

Reference: Gateway API in GKE

BigQuery has two main ways to pay for queries.

• On-demand: you pay for the number of bytes each query scans. Simple, and good for occasional or unpredictable usage. Cost rises directly with careless queries.
• Capacity (editions with slots): you buy a set amount of compute capacity (slots), often with autoscaling, and queries share it. Cost is predictable, and it suits heavy, steady workloads.

Storage is billed separately, and data not edited for 90 days moves to a cheaper long-term rate automatically.

How to choose: start on-demand. If your monthly query bill is large and steady, model the same workload on capacity pricing. Also use reservations to split capacity between teams so one heavy job does not starve the others.

Whichever you pick, partitioning, clustering, and selecting only needed columns reduce cost.

Reference: BigQuery pricing

There are three main tools, and they can be combined.

• Authorized views: create a view that selects only the allowed columns or rows, then authorize that view to read the source table. Users get access to the view, not the base table.
• Column-level security: tag sensitive columns with policy tags (Data Catalog taxonomy). Only users with the fine-grained reader role can see those columns.
• Row-level security: create row access policies so each user sees only their rows, for example only their region's data.

Use an authorized view for a quick, flexible answer. Use policy tags and row policies when you need rules that apply everywhere the table is used.

The trap answer is "copy the data into another table". That works, but creates stale copies, extra cost, and more places to secure.

Reference: Introduction to column-level access control

All three prepare data for reuse, but they behave differently.

• View: a saved query. It stores no data and runs the full query every time you read it, so you pay for what it scans.
• Materialized view: stores the precomputed results and keeps them up to date automatically. Reads are faster and cheaper, and BigQuery can use it to speed up other queries. It supports a limited set of SQL.
• Scheduled query: runs your query on a schedule and writes the result to a regular table. It is flexible, but the data is only as fresh as the last run.

Choose a view for simple logic or access control, a materialized view for repeated aggregations that need to stay fresh, and a scheduled query when you need full SQL freedom and can accept some delay.

Reference: Materialized views

A log sink routes a copy of log entries that match a filter to a destination: Cloud Storage (cheap archive), BigQuery (analysis), Pub/Sub (send to another tool), or another log bucket. Sinks can be created at the project, folder, or organization level, and an organization-level aggregated sink collects logs from every project.

Cost comes mainly from the volume of logs you ingest and keep. Ways to control it:

• Add exclusion filters to drop noisy, low-value logs (for example, health check requests).
• Route high-volume logs to a cheaper destination instead of keeping them in the default bucket.
• Shorten retention where the rules allow.
• Be careful turning on Data Access audit logs everywhere, since they can be very large.

Do not exclude logs you need for security or compliance. Exclude only what you are sure you will never use.

Reference: Routing and storage overview

Start by asking the cluster autoscaler why. Check the autoscaler status and events (kubectl get events, or the GKE console's autoscaler view). Scale-down is cautious by design and has documented reasons for refusing.

The most common causes:

1. A pod on the node cannot be evicted. Bare pods (not owned by a Deployment, ReplicaSet, or Job), pods using local storage, pods whose PodDisruptionBudget would be broken, and pods with the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "false" all block draining.
2. One stuck pod pins the whole node. If any pod cannot be rescheduled elsewhere, the node stays.
3. The node pool minimum is too high. Check this before assuming a scheduling problem.
4. System pods or add-ons without a disruption budget, or marked not safe to evict.

Fix the blocking pod directly: add a controller, adjust the PodDisruptionBudget, move local data to proper storage, or mark the pod safe to evict. The autoscaler is not buggy here. It is refusing to disrupt a workload it cannot safely move.

Reference: Cluster autoscaler

The shape is: a global external Application Load Balancer in front, Cloud Run (or a regional MIG) in at least two regions, and a database chosen by how strict the availability need really is.

The load balancer's single anycast IP and health checks send each user to the nearest healthy region. Because the app tier is stateless, any region can serve any user. So the database is the real decision:

• Cloud SQL with cross-region read replicas: simple. Writes go to one primary region and reads can be local. If the primary region fails, writes are down until you manually promote a replica. That is not instant or automatic.
• Cloud Spanner: one logical database that is multi-region, with automatic failover and no manual promotion. It costs more and needs careful schema design.

In an interview, name the RPO (how much data you can lose) and RTO (how long you can be down) first. Spanner solves a problem replicas cannot, but many services do not need to pay for it.

Reference: Architecture Framework: reliability

Start with the INFORMATION_SCHEMA.JOBS view (or the billing export), sorted by total_bytes_billed, for the time window of the spike. A cost spike is almost always a few expensive queries, not every query getting slightly worse.

Usual culprits:

• SELECT * on a big table, or a missing partition filter on a partitioned table. This forces a full scan. It is the most common cause.
• A scheduled query or dashboard refresh that now runs more often, or whose table grew a lot.
• A new table with no partitioning or clustering that a frequent query now hits.
• A pricing or reservation change, such as a switch between on-demand and capacity pricing, or a changed slot count.

Fixes: set require_partition_filter on the table so every query must filter by partition, select only the columns needed, and move repeated heavy queries onto a materialized view.

The preventive half matters too: set per-query or per-project bytes billed limits so a runaway query fails instead of becoming a bill.

Reference: Control costs in BigQuery

Work through the layers in order, because "permission denied" can come from several independent places.

1. Read the actual error. A 403 naming storage.objects.get is an IAM problem. A message about an organization policy or a perimeter points elsewhere.
2. Check the IAM binding still exists. Run gcloud storage buckets get-iam-policy gs://BUCKET and check the project policy. Someone may have overwritten it (for example, Terraform applied from a stale file).
3. Look for a new Organization Policy, such as domain-restricted sharing. These apply org-wide and can be changed by people unrelated to your pipeline, a classic cause of "it worked yesterday".
4. Check the credential. A rotated or disabled service account key, or a changed Workload Identity Federation setup, gives this exact symptom.
5. Check VPC Service Controls. A perimeter change elsewhere can silently block cross-project access that used to work.

The fastest way to learn what changed is Cloud Audit Logs. Search Admin Activity logs for setIamPolicy or policy update events around the time it broke, and they show who changed what and when.

Reference: Troubleshoot access

Ask what the monolith needs that Cloud Run cannot give. Cloud Run is the lower-effort path when it fits.

Cloud Run is enough when:

• The app is stateless, or state can move to managed services (Cloud SQL, Memorystore, Cloud Storage).
• It talks HTTP or gRPC.
• The team wants scale-to-zero and no cluster to run.
• Deploys are a single container, with no sidecars or init containers.

GKE is the real need when:

• It must run several tightly coupled containers as one unit.
• It needs persistent local state (StatefulSets, specific volume types).
• It needs custom networking: a service mesh, fine network policies, or protocols Cloud Run handles poorly.
• It depends on Kubernetes extras like CRDs or operators.
• Traffic is high and steady, so committed capacity beats pay-per-request cost.

The trap is choosing GKE because it feels more "production grade". For a stateless HTTP app with no Kubernetes dependencies, GKE adds cluster upgrades, node management, and more moving parts for features you will never use. The strong answer names the specific requirement that forces GKE.

Reference: Choosing a compute option

An RPO of 5 minutes means you can lose at most 5 minutes of writes. Scheduled backups alone (daily, maybe hourly) cannot do that. You need continuous replication.

The mechanism is a cross-region read replica plus point-in-time recovery (which needs transaction logging on). In a regional failure:

1. The primary region fails and writes stop.
2. You promote the cross-region replica to be the new primary. In Cloud SQL this is a deliberate manual step. That is good, because an automatic failover during a short network blip could cause split-brain, which is worse than the outage.
3. Replication lag decides real data loss. Cross-region replication is asynchronous. Normally lag is seconds, inside 5 minutes, but it is not guaranteed, so monitor lag and alert when it grows.
4. Point the application at the new primary (update connection strings, DNS, or the Cloud SQL Auth Proxy target).

For a fast RTO as well, keep the app tier already deployed (idle or warm) in the DR region. Standing up compute and promoting the database one after the other is much slower.

If the requirement were an RPO of zero, asynchronous replication cannot promise it. That is the line where the answer becomes Cloud Spanner with synchronous multi-region replication.

Reference: Cloud SQL high availability

Shared VPC alone does not isolate teams. Firewall rules live in the host project, and without more scoping a Shared VPC admin there can see and edit all of them. Isolation comes from combining Shared VPC with hierarchical firewall policies and narrow IAM roles.

The design:

1. One host project and one VPC, with a subnet (or several) per team or environment.
2. Hierarchical firewall policies at the folder level, scoped so each team's rules apply only to its own subnet ranges or tags. This prevents "team A's rule matches team B's VMs" without relying on people writing careful CIDR ranges.
3. Narrow IAM. Only the central network team gets roles/compute.networkAdmin. Each team gets roles/compute.networkUser on its own subnet, not on the whole project.
4. No team gets networkAdmin, since it can change firewall rules and routes for everyone.

The detail people miss: networkUser can be granted at the subnet level. Granting it project-wide on the host project defeats the isolation Shared VPC is meant to give.

Reference: Shared VPC provisioning

They work at different layers and do not know about each other. VPC firewall rules control traffic between VMs (the GKE nodes) by IP, tag, or service account. Kubernetes NetworkPolicies control traffic between pods, enforced inside the cluster (on GKE, by Dataplane V2's eBPF engine, or older Calico-based enforcement).

A VPC firewall rule cannot say "pod A may talk to pod B but not pod C", because from the VPC's view traffic goes to a node, not a pod. That distinction exists only inside the cluster.

You usually need both:

• VPC firewall rules are the outer boundary: what can reach the nodes from outside, and what the nodes can reach outside (an on-prem database, the internet).
• NetworkPolicies are the inner boundary: which pods can talk to which, such as default-deny between namespaces so a compromised pod cannot freely reach a database pod.

Skipping NetworkPolicies because "the firewall already restricts access" is a serious gap. The firewall keeps outsiders out, but it does nothing about lateral movement once something inside is compromised.

Reference: GKE network policy

Occasional 503s under moderate load usually mean requests arrive faster than the service can accept them with its current concurrency and instance settings. That is a scaling configuration problem, not necessarily broken code. Check Cloud Run settings first.

1. maxInstances. If it is too low, Cloud Run cannot scale out and rejects requests once the queue fills. This is the most common cause.
2. Concurrency per instance. If concurrency is 80 but the container can really handle a handful at once before CPU or memory runs out, Cloud Run keeps sending traffic to an overloaded instance. Lower it to match reality.
3. minInstances and cold starts. With minInstances at 0 and bursty traffic, new instances may not start fast enough, so requests time out during ramp-up. A small minimum removes the gap.
4. Request timeout versus downstream speed. A slow database or API under the same load can cause timeouts that look like Cloud Run failures.
5. CPU allocation. With "CPU only during requests", background work and concurrent requests can starve each other.

The telltale sign of a configuration cause: errors are load-dependent and intermittent, not constant. The fix is usually in the service's scaling settings rather than the code.

Reference: About instance autoscaling

For minimal downtime, do one bulk historical load, then keep the target in sync with change data capture (CDC) until cutover. Avoid a single big-bang export that freezes the source for hours.

How it works: Datastream is GCP's managed CDC service. It reads the source change log (binlog for MySQL, WAL for PostgreSQL) and streams inserts, updates, and deletes into BigQuery. In parallel, a one-time bulk export (bq load, Dataflow, or a native export to Cloud Storage) loads history from before the stream began.

When the stream has caught up (check by comparing row counts or checksums), cutover is short: pause writes briefly, let the last changes flush, confirm both sides match, then point consumers at BigQuery. Downtime is minutes, not days.

Name these details in an interview: CDC needs logging enabled on the source before you start, and data types that do not map cleanly to BigQuery must be solved in the design, not discovered during cutover.

Reference: Datastream overview

Start with structure: separate projects per environment (often per team too), grouped under folders like dev/, staging/, prod/. IAM and Organization Policy apply at the project or folder level, so separation must exist in the hierarchy before it can exist in access control.

IAM layer:

• Broad roles at the dev folder, because speed matters and the blast radius is small.
• Narrower roles in staging and prod. In prod most engineers get read-only or no standing access, with elevated access through a break-glass process or time-limited IAM Conditions.
• CI/CD service accounts get exactly what their pipeline needs, with a different account per environment.
• Grant roles to Google Groups, not to individuals, so access follows team membership.

Organization Policy layer (limits what is possible, no matter who asks):

• Restrict resource regions per folder when compliance requires it.
• constraints/iam.disableServiceAccountKeyCreation to force short-lived credentials.
• constraints/compute.vmExternalIpAccess in prod to stop internet-exposed VMs.
• Domain-restricted sharing so resources cannot be shared outside the organization.

The key line: IAM says whether this identity can do this action. Organization Policy says whether this action is allowed in this part of the hierarchy at all. A prod folder that relies only on IAM to stop public buckets is one wrong role binding from an incident.

Reference: Organization Policy overview

Encryption in transit, mutual authentication (mTLS), and auditing, all without code changes, is what a service mesh is for. On GKE the standard answer is Cloud Service Mesh (Google's managed, Istio-based mesh). It adds a sidecar proxy next to each pod so the app does not need to implement TLS or authentication.

How it meets each requirement:

• Encryption without code changes: the sidecar intercepts all traffic and enforces TLS. The app still makes a plain call to the service name.
• Mutual authentication: the mesh gives each workload a certificate identity and enforces mTLS, so both sides verify who they talk to. This is stronger than a NetworkPolicy, which only limits which pods may send traffic, not whether the sender is who it claims to be.
• Auditing: all traffic flows through the sidecar, so every service-to-service call can be logged to Cloud Logging and Monitoring.

The trade-off: a mesh adds operational complexity and a little latency per hop. It is right when you need all three together. For one requirement alone there are lighter options (NetworkPolicies for restriction, an internal load balancer for TLS), but none gives mTLS and per-call audit logs without code changes.

Reference: Cloud Service Mesh overview

IAM alone cannot do this. A user with valid permissions can copy data to a project they own. The tool built for this is VPC Service Controls, which creates a service perimeter around projects and services so data cannot cross the boundary.

Design:

1. Put the sensitive projects in a perimeter and restrict the services involved (BigQuery, Cloud Storage, and others).
2. Block API calls across the boundary. Even with correct IAM, copying data from inside the perimeter to a project outside it is denied.
3. Add ingress and egress rules for the exceptions you genuinely need, for example a trusted CI/CD project or a partner. Keep these as narrow as possible.
4. Use access levels (IP ranges, device or identity conditions) to allow trusted users to reach the perimeter from outside.
5. Test with dry-run mode first. The perimeter logs what it would block, so you can find legitimate flows before enforcing.
6. Back it up with Organization Policies (such as domain-restricted sharing) and audit logs.

The trap is turning it on in enforced mode right away. It is a very common cause of "it worked yesterday" outages, because pipelines that cross projects suddenly get blocked. Dry-run first is the strong answer.

Reference: VPC Service Controls overview

Pub/Sub guarantees at least once delivery, so duplicates can and will happen (a slow ack, a retry, a restart). You cannot prevent every duplicate, so you make the processing safe to repeat. This is idempotency.

Approaches:

1. Use a unique id. Put an id in each message (or use the Pub/Sub messageId). Before doing the work, check a store (such as a database table with a unique key) and skip if it was already handled.
2. Make the operation naturally idempotent. For example, "set status to shipped" is safe to repeat, while "add 1 to counter" is not. Prefer upserts over inserts.
3. Use a transaction so "record that I handled this" and "do the work" succeed or fail together.
4. In Dataflow, use its built-in deduplication on message ids and its checkpointing, and write with an idempotent sink (for example BigQuery with an insert id or a MERGE).
5. Consider exactly-once delivery on a subscription. It reduces redelivery after an ack, but you should still design handlers to be safe, because side effects outside Pub/Sub (an email, a payment call) are not covered.

The common wrong answer is "turn on exactly-once and stop worrying". It narrows the problem but does not remove the need for idempotent side effects.

Reference: Exactly-once delivery

One hot node with the others idle is the classic sign of a hotspot: too many reads or writes go to one range of row keys.

Diagnose:

1. Check Key Visualizer in the console. It shows a heatmap of key ranges over time, and a hotspot appears as a bright horizontal line.
2. Look at the key design. The usual cause is a key that starts with an increasing value, such as a timestamp or a sequential id, so all new writes hit the last tablet.
3. Rule out other causes: a single very large row, a heavy scan from one client, or too few nodes for the load.

Fix:

• Redesign the row key so writes spread out: lead with a high-cardinality value (like device id), or add a salt or hash prefix. Keep your main read pattern in mind, since spreading keys can make range scans harder.
• Fix the access pattern: move heavy scans to a separate app profile or cluster, and break up giant rows.
• Add nodes only if load is truly even. More nodes will not cure a hotspot, because the hot range still sits on one node.

Changing a key means writing the data to a new table, so plan a migration. The honest summary: hotspots are a schema problem first and a capacity problem second.

Reference: Key Visualizer

Spanner splits tables into ranges by primary key. If most writes go to one range, one server does all the work no matter how many nodes you have. This is a hotspot, and the cause is usually the primary key.

Common causes:

• Sequential keys: an auto-increment id or a timestamp as the first key column sends every new write to the end of the table.
• A hot parent row: with interleaved tables, heavy writes under one parent row all land together.
• Lock contention: many transactions updating the same rows (like a global counter) wait on each other.

Fixes:

1. Choose a key that spreads writes: a UUID (version 4), or a bit-reversed sequence. Spanner has a built-in bit-reversed sequence for this purpose. If you need time ordering, put a shard value in front of the timestamp.
2. Break up hot counters into several rows (sharded counters) and add them up when reading.
3. Keep transactions short and touch fewer rows.
4. Use the tools: Key Visualizer, query statistics, and lock statistics show where the pressure is.

The wrong answer is "add more nodes". It helps only when load is already even.

Reference: Schema design best practices

A pod in Pending has been accepted by Kubernetes but not placed on a node yet. The scheduler almost always tells you why. Start here:

Common causes, roughly in order:

1. Not enough resources: no node has the requested CPU or memory. Events say Insufficient cpu or Insufficient memory. Fix by lowering requests, adding nodes, or checking that the Cluster Autoscaler is on and has room to grow (node pool max, quotas, IP range).
2. Taints and tolerations or node selectors: the pod asks for nodes it cannot be placed on (for example a Spot pool taint with no toleration, or a label no node has).
3. Unbound PersistentVolumeClaim: the pod waits for storage that cannot be created or is in another zone.
4. Quota or IP exhaustion: the project quota is used up, or the pod IP range is full.
5. Affinity or topology rules that cannot be satisfied.

If the autoscaler should have added a node but did not, check its events: it often reports a reason such as a quota limit or a pod that no node pool can fit. Always begin with describe pod, because guessing wastes time.

Reference: Troubleshooting scheduling

Workload Identity has several links, and a break in any one gives a permission error. Check them in order.

1. Is Workload Identity enabled on the cluster and the node pool? A node pool without the GKE metadata server will use the node's service account instead, which is a common surprise.
2. Does the pod use the right Kubernetes service account (KSA)? Check serviceAccountName in the pod spec and the namespace.
3. Is the KSA annotated with the Google service account (GSA) email: iam.gke.io/gcp-service-account=GSA_EMAIL?
4. Is the IAM binding in place? The GSA must allow the KSA to impersonate it: the member serviceAccount:PROJECT.svc.id.goog[NAMESPACE/KSA_NAME] needs roles/iam.workloadIdentityUser on the GSA. A typo in the namespace or name is very common.
5. Does the GSA have the role for the API it calls, on the right resource?
6. Check which identity is really used. From inside the pod, call the metadata server (curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email) and see which email it returns.

Step 6 is the fastest: if it returns the node's default account instead of your GSA, the problem is in steps 1 to 4. If it returns your GSA, the problem is its roles (step 5), and Audit Logs will show the denied call.

Reference: Troubleshooting Workload Identity

Keep the database private and connect over the VPC, not the internet.

Steps:

1. Give Cloud SQL a private IP in your VPC (using private services access) and turn off the public IP.
2. Let Cloud Run reach the VPC with Direct VPC egress or a Serverless VPC Access connector, and route private traffic through it.
3. Connect safely. Use the Cloud SQL connector or Auth Proxy, which encrypts the connection and uses IAM to authorize it, so you do not manage IP allow lists or certificates.
4. Use a dedicated service account for the service with roles/cloudsql.client, and consider IAM database authentication so no password is stored.
5. Manage connections. Cloud Run can scale to many instances, and each opens connections, which can exceed the database limit. Use connection pooling, set a sensible maxInstances, and keep concurrency in mind.

The trap is forgetting the connection limit. A traffic spike can scale Cloud Run quickly and use every database connection, so pooling (or a pooler such as PgBouncer or Managed Connection Pooling) matters.

Reference: Connect Cloud Run to Cloud SQL

A landing zone is the base setup that every later project inherits, so teams can move fast without breaking security rules. Build it in layers.

1. Identity. Use Cloud Identity or Google Workspace with SSO and MFA. Manage access with groups, not individuals.
2. Resource hierarchy. An organization with folders for environments or business units (for example prod, non-prod, shared), and a project per workload and environment.
3. Policies at the top. Organization Policies such as: no service account key creation, no external IPs on VMs, allowed regions, domain-restricted sharing, and uniform bucket-level access.
4. Networking. A Shared VPC (or hub-and-spoke) with central firewall policies, Cloud NAT, private access to Google APIs, and a plan for hybrid connectivity (Cloud VPN or Interconnect).
5. Logging and monitoring. An organization-level aggregated log sink to a central, locked-down project, plus Security Command Center.
6. Billing controls. Budgets and alerts, and billing export to BigQuery.
7. Automation. Define all of it as code (Terraform) in Git, applied by a pipeline, so changes are reviewed and repeatable.

The key idea is guardrails, not gates: policies set at the top make the safe path the default, so teams get autonomy inside safe limits. Google publishes a reference blueprint for this, which is a good starting point.

Reference: Enterprise foundations blueprint

Measure first, then cut. Guessing leads to cutting the wrong thing.

1. Find where the money goes.

• Turn on billing export to BigQuery, then group cost by project, service, SKU, and labels.
• Look at the Billing reports for the biggest line items and for sudden jumps.
• Use Recommender insights (idle VMs, oversized VMs, unattached disks, unused IPs).

2. Fix the usual causes.

• Idle or oversized resources: stop unused VMs, right-size machine types, delete old disks and snapshots.
• Compute pricing: cover steady load with committed use discounts, and use Spot VMs for fault-tolerant work.
• Storage: lifecycle rules to colder classes, delete old versions.
• BigQuery: partition and cluster tables, avoid SELECT *, set byte limits.
• Network egress: keep traffic in the same region, use Cloud CDN, and watch cross-region and internet egress.
• Logging: exclude noisy logs.
• GKE: right-size pod requests, use Autopilot or the autoscalers.

3. Prevent it coming back.

• Budgets and alerts per project, and required labels for ownership.
• Review costs regularly (a FinOps habit) and give each team visibility of its own spend.

Interviewers want to hear the order: visibility, then quick wins, then commitments, then guardrails. Buying commitments before cleaning up waste locks in a bigger bill than needed.

Reference: Cost optimization

The goal is to expose a new version to a small share of real traffic, measure it, and only continue if it is healthy.

On Cloud Run

1. Deploy the new revision with no traffic and a tag, and test it on its tagged URL.
2. Shift traffic gradually (for example 5%, 25%, 50%, 100%) using traffic splitting.
3. Watch error rate and latency at each step. If a metric breaks its threshold, send 100% back to the previous revision, which is instant.

On GKE

1. Use a rolling update with readiness probes, so a pod only gets traffic when it is ready, and set maxUnavailable low.
2. For canary releases, use Gateway API traffic weights or a rollout tool such as Argo Rollouts.
3. Use PodDisruptionBudgets so updates and node upgrades never remove too many pods.

Automation

• Cloud Deploy can run the pipeline with approval gates, canary strategy, and automatic rollback on failed verification.
• Define a health signal (such as error rate and p95 latency from Cloud Monitoring) that decides promote or roll back.

Do not forget the database. Zero-downtime code needs backward-compatible schema changes. Add a column first, deploy code that works with both, and remove old fields in a later release. A release that needs a breaking schema change cannot be rolled back safely.

Reference: Cloud Deploy overview

Remove every single point of failure: the link, the device, the location, and the router in GCP.

Option 1: HA VPN (up to 99.99% SLA)

• Use an HA VPN gateway, which has two interfaces.
• Create two tunnels (one per interface) to your on-prem gateway or to two gateways.
• Use Cloud Router with BGP so routes are learned dynamically and traffic switches by itself when a tunnel fails.

Option 2: Interconnect (up to 99.99% with the right layout)

• Use at least two connections in two different edge availability domains in one metro for 99.9%, and add a second metro with redundant connections for 99.99%.
• Run BGP sessions over each VLAN attachment to Cloud Router.

Common design for important systems

• Interconnect as the main path, with HA VPN as a backup over the internet.
• Use BGP priorities (MED) so the preferred path is used normally and the other takes over on failure.
• Consider global dynamic routing so a Cloud Router can advertise routes to all regions of the VPC.

Test it. Actually fail a tunnel or a link in a maintenance window and watch traffic move. Many setups look redundant on paper but share a hidden single point, such as one on-prem router or one fibre path.

Reference: HA VPN topologies

The standard shape is Pub/Sub for ingestion, Dataflow for processing, and BigQuery for storage and analysis.

Key design points:

1. Ingest with Pub/Sub. It absorbs bursts and decouples producers from the pipeline.
2. Process with Dataflow. Parse and validate events, then aggregate.
3. Handle late data. Use event time (when it happened), not arrival time. Use windows (for example 1 minute), a watermark, and allowed lateness so late events still update the right window.
4. Handle bad data. Send records that fail parsing to a dead-letter output (a Pub/Sub topic or a BigQuery table) so one bad event never stops the pipeline, and you can fix and replay them.
5. Avoid duplicates. Deduplicate on an event id, and write to BigQuery with the Storage Write API (supports exactly-once semantics) or a MERGE.
6. Model the BigQuery tables. Partition by event date and cluster by common filters to control cost.
7. Operate it. Alert on backlog age (oldest unacked message) and Dataflow lag, and use autoscaling.

The point to stress: in streaming, data is never clean or on time, so late and bad data handling is part of the design from the start, not an afterthought.

Reference: Streaming pipelines with Dataflow

These two symptoms often feed each other: slow queries hold connections longer, connections pile up, and the database gets even slower. Look at both the queries and the connections.

1. Confirm the picture.

• Check Cloud Monitoring for CPU, memory, and active connections.
• Use Query Insights to find the queries using the most time and CPU.
• Run SELECT * FROM pg_stat_activity; to see who is connected and what they are running, including long-running or idle-in-transaction sessions.

2. Fix the queries.

• Add missing indexes, check EXPLAIN ANALYZE, and remove N+1 query patterns.
• Cancel or fix very long transactions, since they hold locks and connections.

3. Fix the connections.

• Use connection pooling in the app, or a pooler (PgBouncer or Managed Connection Pooling), so many app instances share a small number of real connections.
• Check how many app instances exist. Autoscaling (Cloud Run, GKE) multiplies connections per instance, so cap maxInstances or the pool size per instance.
• Set sensible idle timeouts.

4. Then consider capacity.

• Add read replicas and send read-only traffic to them.
• Scale up the instance size. Raising max_connections alone usually makes things worse because each connection uses memory.

The common wrong answer is "just raise max_connections or add CPU". That hides the cause. Fix the slow queries and pool the connections first, then scale if the load is still real.

Reference: Query Insights