Mid-Level DevOps Engineer Interview
- 2-5 years experience - Roles: DevOps Engineer, Site Reliability Engineer (SRE) - 70 checklist questions, 30 real interview Q&A, 10 live scenarios, 10 behavioral questions - Companies: Atlassian, Stripe, PhonePe, Cloudflare, Grab, Databricks
What You'll Learn
Before You Read This
You cleared the Junior round. You can explain what a pod is, you know how Docker layers work, and you can write a basic Jenkins pipeline. That is table stakes here - nobody at a mid-level interview will ask you to define Kubernetes. What changes at mid-level is what you are expected to own. Junior engineers execute. Mid-level engineers design, troubleshoot systems they did not build, make tradeoff decisions, and explain why they made them. The questions shift from "what is X" to "you have a production problem, walk me through it" and "your team wants to do X, how do you architect it and what breaks?" This module has four tiers. Every question across all tiers is numbered continuously so you always know exactly where you are. Tier 1 - Fundamentals Checklist (no answers) 70 questions across Kubernetes, CI/CD, IaC, Observability, Networking/Security, Docker, GitOps, and Incident Management. These have no answers written out. If you cannot answer any of them cold, go back to the relevant module first. These are the floor - the minimum expected of every mid-level candidate before the interview starts. Tier 2 - Real Interview Questions (Q1 to Q30) 30 questions at mid-level depth with full answers - the kind asked at companies like Atlassian, Cloudflare, Stripe, PhonePe, Grab, and well-funded product startups. Every answer explains what the interviewer is actually testing and what separates a good answer from a mediocre one. Topics: CI/CD architecture, Kubernetes operations, SRE and observability, Terraform, security, Docker, and AWS. Tier 3 - Scenario Round (Q31 to Q40) 10 open-ended production scenarios. No single correct answer exists - the interviewer is watching how you think, prioritize, and communicate under pressure. These replicate the live whiteboard or screen-share rounds used at most mid-market and enterprise companies. Silence is what eliminates candidates here, not wrong answers. Behavioral Round (Q41 to Q50) 10 behavioral questions with guidance on what a strong answer looks like. Mid-level behavioral rounds test ownership, tradeoff thinking, mistake handling, and cross-team communication - not just technical execution. Every question here has appeared in real mid-level loops. ---
Tier 1 - Mid-Level Fundamentals Checklist
> If you need to look any of these up, go back to the relevant module first. These are the floor, not the ceiling. Kubernetes - operational depth * What is the difference between a Deployment, StatefulSet, and DaemonSet? When do you use each? * What is a PodDisruptionBudget and when does it matter? * How does the Kubernetes scheduler decide which node to place a pod on? * What is the difference between a ClusterIP, NodePort, and LoadBalancer service? * How do you configure resource requests and limits, and what happens when a container exceeds its memory limit? * What is a HorizontalPodAutoscaler and what metrics can it scale on? * What is the difference between a ConfigMap and a Secret? * How does RBAC work in Kubernetes? What is the difference between a Role and a ClusterRole? * What is an Ingress and how does it differ from a Service? * What are taints and tolerations used for? * What is the difference between a liveness probe and a readiness probe? * What happens to a pod when its node is drained? * How does Kubernetes handle rolling updates - what do maxSurge and maxUnavailable control? * What is an init container and when would you use one? * What is the purpose of a NetworkPolicy in Kubernetes? CI/CD - architecture level * What is the difference between a pipeline trigger on push vs on pull request? * How do you handle database migrations in a CI/CD pipeline? * What is a monorepo and what challenges does it create for CI/CD? * How do you prevent secrets from leaking in CI logs? * What is a pipeline artifact and how do you share it between stages? * What is the difference between a canary deployment and a blue/green deployment? * How do you implement an automatic rollback in a CI/CD pipeline? * What is a deployment gate and when do you use one? * How do you version Docker images in a pipeline - what tagging strategy do you use and why? * What is the difference between a static code analysis tool and a DAST tool in a pipeline? * What is dependency caching in CI and how does it work? Infrastructure as Code * What is Terraform state drift and how do you detect it? * What is the difference between terraform destroy and terraform apply with a removed resource? * How do Terraform modules work and why do you use them? * What is the difference between count and for_each in Terraform? * What is a Terraform workspace? * What is the difference between terraform import and writing a resource from scratch? * What is a data source in Terraform and when do you use it instead of a resource? * How do you handle sensitive values like passwords in Terraform without committing them to Git? Observability * What is the difference between a metric, a log, and a trace? * What is an SLO and an error budget? * What is cardinality in Prometheus and why does high cardinality cause problems? * What is a histogram metric and when is it more useful than a gauge? * What is the difference between a counter and a gauge in Prometheus? * What is AlertManager and what does it do that Prometheus alone cannot? * What is a recording rule in Prometheus and when do you write one? * What is OpenTelemetry and why does it matter for tracing? Networking and Security * What is mTLS and why is it used in microservices? * What is a service mesh and what problem does it solve? * What is the difference between authentication and authorization? * What is OIDC and how does it work with Kubernetes? * How does a VPN differ from a bastion host for accessing private infrastructure? * What is the difference between a Security Group and a NACL in AWS? * What is a VPC endpoint and when do you use one instead of routing traffic through NAT Gateway? * What is the principle of least privilege and how do you apply it to IAM roles? * What is IRSA (IAM Roles for Service Accounts) and why is it better than using node-level IAM roles? Docker and Containers * What is the difference between a Docker image layer and a container layer? * What is a multi-stage Docker build and what problem does it solve? * What does .dockerignore do and why does it matter for build performance? * What is the difference between CMD and ENTRYPOINT in a Dockerfile? * What happens to data written inside a container when the container is removed? * What is the difference between a bind mount and a named volume in Docker? * How do you reduce Docker image size - name three concrete techniques? GitOps * What is the difference between push-based and pull-based deployment? * How does ArgoCD know when a deployment is out of sync? * What is Flux and how does it differ from ArgoCD? * What is a GitOps reconciliation loop and what triggers it? * How do you handle secrets in a GitOps workflow where everything is committed to Git? Incident Management and Reliability * What is MTTR and MTBF? Which one matters more for user experience? * What is a postmortem and what makes one blameless? * What is the difference between a P1 and P2 incident - how does your response differ? * What is a runbook and how does it differ from a playbook? * What is chaos engineering and what problem does it solve? * What is alert fatigue and what causes it? * How do you decide what severity level to assign an alert? ---
Tier 2 - Real Interview Questions (Q1-Q10)
CI/CD - Architecture and Decisions 1. Your team has a monorepo with 12 microservices. The CI pipeline builds and tests all 12 on every commit. It takes 40 minutes. How do you fix this? What the interviewer is testing: Can you solve a real pipeline architecture problem - not just say "make it faster." This is one of the most common senior/mid-level screen questions. Answer: The first thing to say in an interview: "Before changing anything, I would measure where the time is actually going." Guessing wastes engineering time. bash Look at stage-by-stage timing in your CI provider (GitHub Actions, GitLab CI) Most CI tools show per-step duration in the pipeline run logs Typically one of three things is responsible for 70%+ of the time: 1. Dependency installation (npm ci, pip install, go mod download) 2. Test execution (usually the biggest) 3. Docker image builds Once you know the bottleneck, the fix depends on what it is. For a monorepo specifically, the architectural solution is path-based change detection - only build and test the services that actually changed. Git diff between current commit and last successful build | v Identify which directories changed | v Map directories to service names | v Only trigger CI jobs for affected services In GitHub Actions this looks like: yaml Use paths-filter to detect which services changed - uses: dorny/paths-filter@v2 id: changes with: filters: | payments: - 'services/payments/' orders: - 'services/orders/' Only run payments tests if payments code changed - name: Test payments service if: steps.changes.outputs.payments == 'true' run: cd services/payments && npm test Beyond path detection, add these in order of impact: Cache dependencies aggressively. If package-lock.json or go.sum has not changed, restore the cache instead of downloading again. This is often a 3-5 minute saving per service. Parallelize across services. Run the CI jobs for affected services simultaneously, not sequentially. Most CI systems support matrix builds or parallel job graphs. Separate fast gates from slow gates. Lint and unit tests should run on every push and complete in under 3 minutes. Integration tests, end-to-end tests, and security scans should only run on PRs to main. Use Docker layer caching. Structure your Dockerfile so the dependency installation layer is cached separately from the source code layer. The dependency layer only invalidates when the lock file changes. dockerfile Copy lock files first - this layer caches as long as deps are unchanged COPY package.json package-lock.json ./ RUN npm ci Copy source code after - this layer rebuilds on every code change COPY src/ ./src/ RUN npm run build Done well, a 40-minute pipeline for a 12-service monorepo can come down to 6-8 minutes for a typical PR that touches 1-2 services. What to say at the end: "I would also talk to the developers about what they actually need fast feedback on. Often teams are running expensive integration tests on every push because nobody questioned the default. Fast feedback on a subset is more valuable than slow feedback on everything." --- 2. You need to implement a CI/CD pipeline that handles database schema migrations safely. What does your pipeline look like? What the interviewer is testing: Database migrations are where most deployment strategies break. This question separates engineers who have done it from those who have only read about it. Answer: The core problem is that migrations and application deployments are two separate things that happen to need coordination. Getting this wrong causes either downtime or data loss. The general rule: migrations must be backward compatible with both the old and new version of the application code. This means you cannot add a NOT NULL column without a default in the same deploy as the code that writes to it - because during the rollout window, old pods are still running and they do not write that column. A safe migration pipeline looks like this: Stage 1: Run migration in a transaction | v Stage 2: Verify migration succeeded (check schema version) | v Stage 3: Deploy new application version | v Stage 4: Run smoke tests | v Stage 5 (optional): Run cleanup migration to remove old columns Breaking a migration across multiple deploys: Deploy 1 - additive only: sql -- Safe: add column with a default, nullable, or backfilled ALTER TABLE orders ADD COLUMN status_v2 VARCHAR(50) DEFAULT 'pending'; New application code writes to both status and status_v2. Old code still writes only status. Nothing breaks. Deploy 2 - switch reads: Application now reads from status_v2. Old code reading from status has been fully replaced. Deploy 3 - cleanup: sql -- Safe to drop old column now that no code references it ALTER TABLE orders DROP COLUMN status; In the pipeline itself: yaml Migration stage runs before app deployment - name: Run database migration run: | ## Use a migration tool with version tracking (Flyway, Liquibase, golang-migrate) flyway migrate -url=$DATABASE_URL ## Fail the pipeline if migration fails - never deploy an app to a broken schema - name: Verify migration version run: | flyway info -url=$DATABASE_URL | grep "Success" > ⚠️ Security: Never put the database connection string directly in pipeline YAML. Pull it from a secrets manager at runtime. The question interviewers follow up with: "What do you do if a migration runs successfully but the app deployment fails?" Answer: Your migration is committed. The database is in the new schema. Rolling back the app is safe only if the migration was backward compatible - which it should be by design. If it was not, you now have a problem. This is exactly why the expand-then-contract pattern (three deploys above) exists. --- 3. What is GitOps and how is it architecturally different from a standard push-based CI/CD pipeline? What the interviewer is testing: GitOps is a 2026 filter question. Companies running Kubernetes at scale have almost universally moved toward it. Knowing the topology is not enough - you need to explain the security model. Answer: In a standard push-based pipeline, the CI system (GitHub Actions, Jenkins) has credentials to your Kubernetes cluster and deploys directly. Developer pushes code | v CI pipeline builds image, runs tests | v CI pipeline runs: kubectl apply or helm upgrade | v <- CI system needs cluster credentials Kubernetes cluster is updated In a GitOps pull-based model, the CI system has no cluster credentials at all. Instead, an operator running inside the cluster watches a Git repository and applies whatever is declared there. Developer pushes code | v CI pipeline builds image, pushes to registry | v CI pipeline updates image tag in Git (the "desired state" repo) | v ArgoCD/Flux detects Git has changed | v <- cluster reaches OUT to Git, not the other way Operator applies changes to the cluster from inside Why this matters from a security standpoint: In the push model, if your CI system is compromised - a malicious pipeline, a stolen token, a supply chain attack - the attacker has direct access to your production Kubernetes cluster. Every build agent has cluster admin credentials sitting in environment variables. In the pull model, no external system has cluster credentials. The cluster connects out to Git, reads what it should look like, and reconciles. An attacker who compromises your CI pipeline can push a bad image to the registry, but they cannot directly execute commands against your cluster. ArgoCD specifically: ArgoCD runs as a controller inside your cluster. It compares the live cluster state to what is declared in your Git repo every few minutes (configurable). If they diverge - because someone ran kubectl apply manually, or a deployment was edited directly - ArgoCD marks the application as OutOfSync and can automatically reconcile it back to what Git says. bash Check sync status argocd app get my-application Manually sync if auto-sync is off argocd app sync my-application See what changed (drift detection) argocd app diff my-application > 📌 Remember: GitOps does not replace CI. CI still builds images and runs tests. GitOps only replaces the deployment step. The two work together - CI produces the artifact, GitOps deploys it. --- Kubernetes - Operational Depth 4. A service is responding to 97% of requests normally but 3% are timing out. Health checks are passing. How do you diagnose this? What the interviewer is testing: This is the most sophisticated Kubernetes debugging question at mid-level. It specifically tests whether you understand that health checks passing does not mean the service is healthy. Answer: The key insight here is that health checks only test if the pod is alive, not if it is working correctly for all requests. A pod with a memory leak or a deadlocked thread pool can pass a health check and still fail real traffic. Step 1 - Isolate whether the issue is infrastructure or application bash Check if the errors are concentrated on specific pods kubectl get pods -n production -l app=payment-service Note the pod names Check per-pod error rates in your metrics In Prometheus: rate(http_requests_total{status=~"5..",job="payment-service"}[5m]) by (pod) If one pod shows 30% errors while others show 0%, that pod is sick If the errors are concentrated on one pod, that pod has a problem the health check is not catching - often a thread pool exhaustion or a broken database connection that was established before the check ran. Step 2 - Check if it is a connection draining issue A 3% failure rate on a 3-replica deployment strongly suggests one pod is unhealthy. The Service load balances roughly evenly across 3 pods - if one pod is rejecting 100% of its requests, you see approximately 33% failures. If it is rejecting ~10%, you see ~3% overall. This is a classic signature. bash Check if a pod is consistently slow or failing kubectl top pods -n production -l app=payment-service Look for a pod with noticeably higher CPU or memory Check logs specifically on the suspected pod kubectl logs <specific-pod-name> -n production --tail=200 | grep -i "error\|timeout\|exception" Step 3 - Check connection pool exhaustion This is one of the most common causes. The application has a fixed pool of database connections. Under load, if connections are not returned quickly (slow queries, long transactions, connection leaks), new requests queue waiting for a connection, then time out. bash Inside the pod - check active connections to the database kubectl exec -it <pod-name> -- netstat -an | grep :5432 | grep ESTABLISHED | wc -l If this number is at or near your max_connections, the pool is exhausted Step 4 - Check for node-level issues bash Is the node the pod is running on under pressure? kubectl describe node <node-name> | grep -A5 "Conditions:" Look for MemoryPressure, DiskPressure, or NetworkUnavailable Is the CNI (network plugin) dropping packets? kubectl get events -n production | grep -i "network\|timeout" Step 5 - Use distributed traces If you have Jaeger or Zipkin set up, look at the traces for the failing 3% of requests. The trace will show you exactly where in the call chain the timeout is happening - whether it is the service itself, its database call, or a downstream service it is calling. What to say in the interview: "I would not immediately restart the pod. I would first capture the state - which pod, what its resource usage is, what its connection count is - because once you restart, the evidence is gone. Capture first, fix second." --- 5. Explain how HPA and VPA work and when you would use each. What happens if you use both at the same time? What the interviewer is testing: Scaling judgment. Many engineers know what HPA and VPA are but have not thought through the conflict. Answer: HPA (HorizontalPodAutoscaler) scales the number of pod replicas based on observed metrics. The default metric is CPU utilization, but you can scale on any custom metric via the metrics API. Traffic increases -> CPU rises -> HPA adds more replicas Traffic decreases -> CPU drops -> HPA removes replicas yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: payment-service-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: payment-service minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 ## scale up when average CPU hits 70% VPA (VerticalPodAutoscaler) adjusts the CPU and memory requests/limits of individual pods over time, based on observed usage. If you set requests: memory: 256Mi but the container consistently uses 600Mi, VPA will update the pod spec to request 600Mi. VPA is most useful when you have a pod where traffic does not vary much but the right resource size is hard to know upfront - like a batch job, a sidecar, or a database. The conflict when using both: HPA scales replicas based on CPU utilization relative to the resource request. VPA changes the resource request. If VPA increases the memory request, Kubernetes needs to restart the pod (pod spec changes require a restart). That restart temporarily removes a replica, which might trigger HPA to add another. The two controllers fight each other. The official guidance (and what you should say in an interview) is: do not use HPA and VPA on the same resource metric at the same time. You can use HPA on CPU and VPA on memory - different metrics - and they coexist without conflict. Or use VPA in Off or Initial mode just to get recommendations without auto-applying them. > 📌 Remember: HPA is for stateless services where you handle load by running more copies. VPA is for pods where you need to right-size the resource allocation. Pick one based on the problem you are solving. --- 6. What is a PodDisruptionBudget and what breaks if you do not use one? What the interviewer is testing: Operational maturity. Many engineers deploy to Kubernetes without PDBs and have never had a node maintenance event to teach them why it matters. Answer: A PodDisruptionBudget tells Kubernetes the minimum number of pods of a given application that must be available at any time during a voluntary disruption - a node drain, a cluster upgrade, or a rolling deployment. Without a PDB, when you run kubectl drain <node> to take a node offline for maintenance, Kubernetes will evict every pod on that node as fast as it can. If all 3 replicas of your payment service happen to be on that node, all 3 are evicted simultaneously. Your service goes down during a maintenance window you planned. yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: payment-service-pdb spec: minAvailable: 2 ## at least 2 pods must be running at all times selector: matchLabels: app: payment-service With this PDB in place, kubectl drain will only proceed when it can guarantee at least 2 payment-service pods remain running. If draining a node would bring the count below 2, the drain pauses and waits for a replacement pod to start on another node first. The tradeoff: A strict PDB can slow down or completely block node maintenance if the cluster does not have enough spare capacity to reschedule the pods. A PDB of minAvailable: 3 on a 3-replica deployment running at full capacity will block drains indefinitely - there is nowhere to move the pods. The right approach is to set minAvailable to the minimum that keeps the service functional, not the maximum you want in normal operation. For a 3-replica service, minAvailable: 2 is usually the right choice. > 🔴 Common Mistake: Setting minAvailable equal to the total replica count. This blocks all maintenance. Always leave at least one disruption slot. --- 7. Walk me through what happens inside Kubernetes when you scale a Deployment from 3 to 10 replicas. What the interviewer is testing: Whether you understand the control plane internals - the reconciliation loop. This separates engineers who have used Kubernetes from those who understand it. Answer: This is a multi-component orchestration that shows how the entire control plane works together. kubectl scale deployment payment-service --replicas=10 | v kubectl sends PATCH request to API Server API Server validates the request and your RBAC permissions API Server writes new desired state (replicas: 10) to etcd | v Deployment Controller (inside kube-controller-manager) is watching etcd for changes to Deployment objects It sees: desired=10, actual=3 -> creates a ReplicaSet with 10 desired | v ReplicaSet Controller sees 7 pods need to be created It creates 7 new Pod objects in etcd (status: Pending) | v Scheduler watches for Pending pods with no node assignment For each pod it evaluates: resource requests, node capacity, taints/tolerations, affinity rules, topology spread constraints It writes a node assignment to each pod object in etcd | v kubelet on each assigned worker node watches etcd for pods assigned to its node It pulls the container image (if not cached) Starts the containers using the container runtime (containerd) Reports pod status back to the API Server | v Endpoints controller sees new Running pods with matching labels Updates the Service Endpoints object so traffic routes to new pods | v kube-proxy on each node updates iptables/ipvs rules New pods now receive traffic The key insight to mention in an interview: nothing in Kubernetes calls anything directly. Every component watches etcd for state changes and acts on them. This is why Kubernetes is resilient - if the scheduler crashes and restarts, it just reads current state from etcd and continues where it left off. --- 8. What is the difference between a Deployment rolling update and a blue/green deployment in Kubernetes? When would you choose each? What the interviewer is testing: Deployment strategy judgment. Junior engineers often treat "blue/green" as automatically better - the mature answer explains the real tradeoffs. Answer: Rolling Update (Deployment default): Kubernetes replaces old pods with new pods gradually - a few at a time, controlled by maxSurge and maxUnavailable. yaml strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 ## can temporarily have 2 extra pods above desired count maxUnavailable: 0 ## never have fewer pods than desired count With this config on a 10-replica deployment, Kubernetes starts 2 new pods, waits for them to pass readiness checks, then terminates 2 old pods. Repeats until all 10 are updated. At no point are you below 10 available pods. Pros: No extra infrastructure needed. Traffic shifts gradually - a bad deploy affects a growing percentage of traffic, not 100% at once. Cons: Old and new versions run simultaneously during the rollout. If they are incompatible (different API contracts, different database schemas), this causes errors during the window. Blue/Green: Two complete environments. Blue is live. Green is the new version. You switch 100% of traffic at once. In Kubernetes, you implement this with separate Deployments and a Service that selects by label: yaml Traffic goes here kind: Service spec: selector: app: payment-service version: blue ## change this to 'green' to do the cutover --- Blue deployment - currently live kind: Deployment metadata: name: payment-service-blue spec: template: metadata: labels: app: payment-service version: blue --- Green deployment - new version, warmed up and ready kind: Deployment metadata: name: payment-service-green spec: template: metadata: labels: app: payment-service version: green Pros: Clean cutover. You can test green fully before switching. Rollback is instant - just change the selector back. Cons: Requires double the compute during the transition. Both environments must be running simultaneously. When to choose which: * Use rolling update for most stateless services - it is simpler and uses less infrastructure. * Use blue/green when you have a schema change that makes old and new code incompatible during the transition window, when you need the ability to test the new version under real load before switching, or when instant rollback is a hard requirement. --- Observability and SRE 9. Your team wants to define SLOs for the first time. Walk me through how you would do it for an order service. What the interviewer is testing: Whether you have actually operationalized SLOs or just know the acronym. Companies with mature DevOps practices ask this at mid-level. Most candidates freeze here. Answer: An SLO (Service Level Objective) is a target for how well your service should perform, expressed as a percentage over a time window. It exists so your team has an agreed definition of "working" before an incident happens, not during one. Step 1 - Identify what matters to the user For an order service, the user cares about three things: * Can they place an order successfully? * Does the order page load fast enough? * Is the order data accurate? Translate these into measurable signals - the Google SRE book calls these the four golden signals: latency, traffic, errors, and saturation. Step 2 - Write specific SLOs Availability SLO: 99.9% of order placement requests return a success response over a 28-day rolling window. (This allows ~43 minutes of downtime per month) Latency SLO: 95% of order placement requests complete in under 300ms. 99% complete in under 1000ms. Over a 28-day rolling window. Step 3 - Define the error budget If your availability SLO is 99.9%, your error budget is 0.1% - that is the amount of failure you are allowed per month. Over 28 days: 28 24 60 * 0.001 = ~40 minutes of allowed downtime. The error budget is the key to balancing reliability and velocity. If you have burned 80% of the month's error budget in week 2, that is the signal to slow down feature releases and focus on stability. If you are at 5% burned at month end, you have budget to take on riskier changes. Step 4 - Instrument it in Prometheus yaml Availability: track the ratio of successful requests A request is successful if it returns 2xx or 3xx - record: job:order_service_availability:ratio_rate5m expr: | sum(rate(http_requests_total{job="order-service",status=~"[23].."}[5m])) / sum(rate(http_requests_total{job="order-service"}[5m])) Alert when error budget burn rate is too fast - alert: OrderServiceErrorBudgetBurning expr: job:order_service_availability:ratio_rate5m < 0.999 for: 5m labels: severity: warning annotations: summary: "Order service availability SLO at risk" What to add unprompted in an interview: "I would also define what counts as a valid SLI - which requests should be included. Health check requests from the load balancer should probably not count toward the SLO. A synthetic monitoring probe that hits the endpoint every 30 seconds should count. Getting this boundary right matters more than the math." --- 10. Explain what high cardinality means in Prometheus and why it is a problem. What the interviewer is testing: Real Prometheus operational experience. High cardinality is the most common reason Prometheus clusters run out of memory. Engineers who have not hit it in production often do not know about it. Answer: In Prometheus, every unique combination of label values creates a separate time series. Cardinality is the total number of unique time series being tracked. High cardinality means you have labels that can take a very large number of different values - which creates an explosion of time series. The dangerous labels: python This is fine - region has 5-10 possible values http_requests_total{service="orders", region="ap-south-1", status="200"} This is dangerous - user_id can be millions of values Each unique user_id creates a separate time series http_requests_total{service="orders", user_id="user-8472901", status="200"} This is catastrophic - request_id is unique per request http_requests_total{service="orders", request_id="550e8400-e29b-41d4-a716-446655440000"} One new time series per request = your TSDB fills up and OOMs in minutes Why it matters in production: Prometheus stores all active time series in memory. Each time series uses roughly 3-4KB of memory. If you have 10 million time series (easy to hit with a user_id label on a service with 1 million users), that is 30-40GB of RAM just for the metric storage - before any actual data. When Prometheus runs out of memory, it crashes. Depending on your storage configuration, you may lose recent data. Your alerting goes dark. This is a production outage caused by a monitoring system that was supposed to prevent outages. How to fix it: * Never add high-cardinality values (user IDs, request IDs, session tokens, IP addresses) as label dimensions * If you need per-user data, put it in logs or traces, not metrics * Use recording rules to pre-aggregate high-cardinality data at collection time * Run promtool tsdb analyze periodically to find your highest-cardinality metrics bash Check your current series count curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries' Find the metrics with most series curl -s 'http://localhost:9090/api/v1/label/__name__/values' | jq '.data | length' > 🔴 Common Mistake: Adding environment-specific details like pod names or container IDs as labels without thinking. pod_name sounds useful, but in a 100-pod deployment with rollouts, the pod names change constantly - each rollout creates new time series and orphans old ones. ---
Tier 2 - Real Interview Questions (Q11-Q18)
11. You are designing an observability stack for a new microservices deployment with 30 services. Walk me through your choices and why. What the interviewer is testing: End-to-end observability thinking. Strong candidates get eliminated here when they just name the tools without explaining the selection logic. Answer: Start with what you are trying to achieve, not what tools to install. You need to answer three questions about your system at any time: * Is it working? (metrics + alerting) * Why is it failing? (logs + traces) * Who is affected and how badly? (SLOs + error budgets) Metrics - Prometheus + Grafana Prometheus scrapes metrics from your services on a pull model. Every service exposes a /metrics endpoint. Prometheus reads it on a schedule. For 30 services, you will want: * One Prometheus instance per cluster (or use the operator pattern) * Recording rules to pre-aggregate expensive queries * AlertManager to route alerts to PagerDuty, Slack, or your on-call tool * Grafana for dashboards, pulling from Prometheus as the data source Logs - Loki (if on Grafana stack) or ELK If you are already using Grafana for metrics, Loki is the natural choice for logs - same query interface, same dashboard, and operationally much simpler than ELK. Loki only indexes labels (not the full log content), which makes it far cheaper to run. ELK (Elasticsearch, Logstash, Kibana) is more powerful for full-text search across structured logs but costs significantly more in compute and storage, and Elasticsearch requires careful tuning to stay healthy. bash Loki query - same syntax as PromQL (LogQL) {namespace="production", app="order-service"} |= "error" | json | status >= 500 Traces - OpenTelemetry + Tempo or Jaeger For 30 services, distributed tracing is not optional - it is how you find which service in a chain is causing latency. Without traces, you are guessing. Use OpenTelemetry as the instrumentation standard. It is vendor-neutral - you instrument once and can send to Jaeger, Tempo, Zipkin, or a commercial tool without changing your code. For a self-hosted stack: Tempo (from Grafana) integrates natively with Grafana and Loki, so you can correlate a log line directly with its trace. The integration that makes this valuable: The three pillars become powerful when they connect. In Grafana, you can: 1. See an error rate spike in a metric dashboard 2. Click through to the logs for that time window 3. Find a specific log line with a trace ID 4. Click the trace ID to see the full distributed trace This workflow takes a potential 30-minute debugging session down to 3 minutes. What to say about OpenTelemetry specifically: "I would use OpenTelemetry's auto-instrumentation where possible - for Java, Python, and Go services, it instruments HTTP clients and servers automatically without changing application code. Manual spans are only needed for custom business operations that matter to trace." --- Infrastructure as Code 12. Your terraform apply fails halfway through. What do you do? What the interviewer is testing: Terraform state management under real conditions. This is the 2AM scenario they want to know you can handle. Answer: A partial apply is one of the most stressful IaC situations because your infrastructure is now in an inconsistent state - some resources were created, some were not, and the state file may or may not accurately reflect what actually exists. Step 1 - Do not run terraform apply again immediately Your first instinct might be to just re-run. Do not. Understand what state the apply left things in first. bash See what Terraform currently thinks exists terraform state list Compare state to reality for specific resources terraform plan This will show what Terraform thinks needs to happen to reach desired state Step 2 - Identify what was created before the failure bash Check the apply output logs for the last successful resource creation Or check your cloud console - which resources exist? If a resource was created but Terraform did not record it in state: terraform import aws_instance.web i-1234567890abcdef0 This adds the existing resource to state without creating a new one Step 3 - Decide: fix forward or roll back In most cases, fix forward is safer. You already have partial infrastructure. Trying to destroy it and start over can cause more damage than carefully applying the remaining changes. bash Target only the failed resource to continue terraform apply -target=aws_rds_instance.main This applies changes only to the specified resource and its dependencies If the failure was in a destructive change (a database was being replaced and the old one was deleted but the new one failed to create), you have a data loss situation. This is why you should never do in-place replacements of stateful resources - use blue/green patterns at the infrastructure level too. Step 4 - Communicate status while you work This is what interviewers actually watch for. "I would immediately post in the on-call channel that we have a partial infrastructure state and services may be degraded. Do not deploy anything until this is resolved." Prevention - what you say you would add: hcl Add lifecycle rules to prevent accidental destruction of critical resources resource "aws_rds_instance" "main" { lifecycle { prevent_destroy = true ## Terraform will refuse to destroy this resource } } And always use -out to save the plan and apply that exact plan: bash terraform plan -out=tfplan.out Review the plan carefully terraform apply tfplan.out Applies exactly what you reviewed --- 13. Explain Terraform modules and when they make things worse instead of better. What the interviewer is testing: IaC maturity. Blind use of modules is a junior mistake - knowing when not to use them is a mid-level signal. Answer: A Terraform module is a reusable block of configuration - a folder of .tf files that can be called multiple times with different inputs. Modules enforce consistency and reduce duplication. hcl A module that creates a standard EKS cluster module "eks_cluster" { source = "./modules/eks" cluster_name = "production" cluster_version = "1.29" node_count = 3 instance_type = "t3.xlarge" region = "ap-south-1" } Same module, different inputs for staging module "eks_cluster_staging" { source = "./modules/eks" cluster_name = "staging" cluster_version = "1.29" node_count = 1 instance_type = "t3.medium" region = "ap-south-1" } When modules are the right choice: * Infrastructure patterns that repeat across environments (VPCs, EKS clusters, RDS instances with standard configurations) * Enforcing security baselines - all VPCs must have flow logs, all S3 buckets must be private - by building that into the module * Large teams where different people manage different parts of infrastructure When modules make things worse: Over-abstraction. A module that wraps a single aws_s3_bucket resource adds complexity without value. The module input variables become a reinvention of the resource's own arguments, worse. Version drift. If you are using a module from a public registry (Terraform Registry or GitHub) and you do not pin the version, a module update can change your infrastructure on the next plan. Pin your module sources: hcl module "vpc" { source = "terraform-aws-modules/vpc/aws" version = "5.1.2" ## always pin - never leave this floating } Debugging complexity. When a module fails, the error points into the module's internals. If the module has three levels of nesting, finding the actual resource that caused the error requires unwrapping multiple layers. Inflexibility. A module built for 80% of your use cases will eventually need to handle the 20%. Adding every edge case as a module variable makes the module interface as complex as the underlying resources - except now you have an indirection layer too. The right signal for "should this be a module" is: will I use this exact pattern at least 3 times, and does it have meaningful internal logic that should be hidden from callers? --- Networking and Security 14. What is mTLS and how does it work in a microservices environment? What problem does it solve that TLS alone does not? What the interviewer is testing: Security depth beyond the basics. mTLS is directly relevant to service mesh adoption and zero-trust architecture, which comes up at any company with mature microservices. Answer: Standard TLS (one-way TLS) is what your browser uses when it visits any HTTPS website. The server presents a certificate. The client verifies it. The client is anonymous. Client -> "I want to connect to api.company.com" Server -> presents certificate (proves it is api.company.com) Client -> verifies certificate, trusts the server Encrypted connection established The problem: In a microservices environment, the server knows it is talking to someone with a valid encrypted connection, but it does not know which service is calling it. Any service - or any attacker who has gained access to the internal network - can call the payment service. mTLS (mutual TLS) adds certificate authentication in both directions. The client also presents a certificate, and the server verifies it. Service A -> presents its own certificate (proves it is Service A) Service B -> presents its certificate (proves it is Service B) Both verify each other's certificates Encrypted connection established with mutual identity proof Now the payment service can enforce: "I will only accept connections from the order service and the auth service. If anything else tries to connect, I reject it at the TLS level before any application code runs." How a service mesh implements this: Manually managing certificates for 30 services is operationally nightmarish - rotating certificates, distributing them to the right pods, handling renewals. This is exactly what service meshes like Istio and Linkerd automate. The mesh injects a sidecar proxy (Envoy in Istio's case) into every pod. The control plane issues short-lived certificates to each sidecar automatically. mTLS between services happens at the sidecar level - your application code does not know it is happening. bash In Istio, enable strict mTLS for an entire namespace kubectl apply -f - <<EOF apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: production spec: mtls: mode: STRICT ## reject any non-mTLS traffic EOF What mTLS does not solve: It proves that Service A has the right certificate, not that Service A is authorized to perform a specific action. For that you still need application-level authorization (checking if the calling service has permission to access a particular endpoint). mTLS is identity, not permissions. --- 15. A developer accidentally committed AWS credentials to a public GitHub repository 10 minutes ago. You get notified. Walk me through your incident response in order. What the interviewer is testing: Security incident response speed and sequence. The correct order matters - most candidates know to revoke the key but do not move fast enough or in the right order. Answer: This is a race. GitHub's ecosystem and malicious bots actively scan public repositories for credentials in near real-time. By the time you are notified at 10 minutes, the key may already have been found. Minute 0-2: Disable the key immediately bash Get the access key ID from the commit (check the git history) git log --patch -- path/to/committed/file | grep -i "AKIA" Disable it - this is faster than deleting aws iam update-access-key \ --access-key-id AKIAIOSFODNN7EXAMPLE \ --status Inactive \ --user-name developer-name Do this before anything else. An inactive key cannot be used even if it was already found. Minute 2-5: Investigate what happened with the key bash Check CloudTrail for any API calls made with this key aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \ --start-time $(date -d "2 hours ago" --iso-8601=seconds) Look specifically for: - New IAM users or access keys created (attacker creating persistence) - EC2 instances launched (cryptomining) - S3 data accessed or exfiltrated - Lambda functions created (attacker infrastructure) - CloudFormation stacks (attacker deploying their own infrastructure) Minute 5-10: Generate a new key, update all systems bash Create a replacement key aws iam create-access-key --user-name developer-name Update every system using the old key: - CI/CD environment variables - Application config (Secrets Manager, Vault) - Local developer machines Then permanently delete the old key (not just disable) aws iam delete-access-key \ --access-key-id AKIAIOSFODNN7EXAMPLE \ --user-name developer-name Remove from GitHub - but understand this does not help security Even after you rewrite Git history to remove the key, GitHub's API indexers have already scanned it. Any bot that found it has the credentials stored. Removal is good hygiene but treat the key as fully compromised regardless of whether you cleaned up the history. After the incident - the prevention layer: bash Add to every developer's pre-commit hook pip install detect-secrets detect-secrets scan > .secrets.baseline This blocks commits containing high-entropy strings that look like credentials Enable GitHub Advanced Security (secret scanning) on the org This scans every push for credential patterns and alerts immediately > 📌 Remember: The correct answer to "how should EC2 instances access AWS services" is IAM Instance Roles, not access keys. This whole incident class is prevented by never using long-lived access keys on infrastructure. Mention this at the end of your answer. --- Docker and Containers - Advanced 16. What is a multi-stage Docker build and why does it matter in production? What the interviewer is testing: Image optimization and production best practices. Every serious DevOps role expects this. Answer: A multi-stage build uses multiple FROM statements in a single Dockerfile. Each FROM starts a new stage. You can copy files from one stage to the next - but only the files you explicitly copy, not everything in the build environment. The problem it solves: build tools (compilers, test frameworks, package managers) are needed to build your application but should never be in your production image. They add size, attack surface, and potential vulnerabilities. Without multi-stage (naive approach): dockerfile FROM golang:1.21 golang:1.21 is 800MB WORKDIR /app COPY . . RUN go build -o server . Final image: 800MB base + your app + all Go toolchain CMD ["./server"] Your production image ships the entire Go compiler, standard library sources, and build tools - none of which your running application needs. With multi-stage: dockerfile Stage 1 - build environment FROM golang:1.21 AS builder WORKDIR /app COPY go.mod go.sum ./ RUN go mod download ## cache dependencies separately COPY . . RUN CGO_ENABLED=0 go build -o server . ## static binary, no external dependencies Stage 2 - runtime environment scratch is literally empty - no OS, no shell, no package manager FROM scratch COPY --from=builder /app/server /server COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ CMD ["/server"] The final image: your binary (~10-20MB) plus CA certificates. Nothing else. No shell, no package manager, no way for an attacker to run arbitrary commands even if they find a vulnerability in your app. For Python or Node.js (interpreted languages where you cannot produce a static binary): dockerfile Stage 1 - install all deps including dev deps for building FROM python:3.11 AS builder WORKDIR /app COPY requirements.txt . RUN pip install --user -r requirements.txt Stage 2 - copy only the installed packages, not the build tools FROM python:3.11-slim COPY --from=builder /root/.local /root/.local COPY src/ ./src/ ENV PATH=/root/.local/bin:$PATH CMD ["python", "src/main.py"] python:3.11-slim is 45MB versus python:3.11 at 1.1GB. The --from=builder copy brings over only the installed packages. What to mention about security: "Smaller images have a smaller attack surface. Every package that is not in the image is a CVE that cannot affect your container. I would run Trivy against the final image in the CI pipeline and block on CRITICAL findings." --- 17. What is the difference between docker stop, docker rm, and what happens to volumes when you remove a container? What the interviewer is testing: Container lifecycle understanding. Volumes are where many engineers lose data by accident. Answer: docker stop sends SIGTERM to the main process, waits a grace period (default 10s), then sends SIGKILL if still running. The container is stopped but its filesystem still exists. You can docker start it again and it resumes from where it was. docker rm deletes the container's writable layer - the filesystem that sits on top of the image layers. This removes all files written inside the container that were not stored in a volume. The container is gone permanently. bash Stop then remove in sequence docker stop my-container docker rm my-container Or stop and remove in one command docker rm -f my-container ## -f force-removes even if running (sends SIGKILL) What happens to volumes: By default, docker rm does NOT remove volumes attached to the container. Named volumes persist until explicitly deleted. bash This removes the container but the 'postgres-data' volume still exists docker rm postgres-container To remove both container and its volumes: docker rm -v postgres-container ## removes anonymous volumes only Named volumes still survive even with -v Explicitly remove a named volume docker volume rm postgres-data Remove all unused volumes (careful in production) docker volume prune The distinction between anonymous and named volumes: dockerfile Anonymous volume - created at container start, random ID VOLUME /var/lib/postgresql/data Named volume - you explicitly create it, it persists by name docker run -v postgres-data:/var/lib/postgresql/data postgres Named volumes are the correct approach for any data you care about. Anonymous volumes are fine for temporary data that does not need to outlive the container. > 🔴 Common Mistake: Running docker system prune without realizing it removes stopped containers and dangling volumes. In production, this can delete database data. Always check docker volume ls before running any prune command. --- AWS and Cloud Architecture 18. Design the networking architecture for a production application on AWS. It needs a web tier, application tier, and database tier with appropriate security boundaries. What the interviewer is testing: AWS networking depth - VPC design, subnet architecture, security groups. This is a whiteboard question that separates junior from mid-level thinking. Answer: The principle is defense in depth - each tier can only talk to the tier it needs, through the ports it needs, and nothing else. Internet | v Internet Gateway | v Public Subnet (AZ-a and AZ-b) - Application Load Balancer (internet-facing) - NAT Gateway (for private subnets to reach internet) | v Private Subnet - App Tier (AZ-a and AZ-b) - EC2 instances / EKS worker nodes - Auto Scaling Group | v Private Subnet - Data Tier (AZ-a and AZ-b) - RDS Multi-AZ (primary in AZ-a, standby in AZ-b) - ElastiCache (Redis) Security groups - the key detail: Security groups are the firewall rules that enforce tier separation. ALB Security Group: Inbound: 443 from 0.0.0.0/0 (internet HTTPS) 80 from 0.0.0.0/0 (redirect to HTTPS) Outbound: 8080 to App SG (forward to app tier) App Tier Security Group: Inbound: 8080 from ALB SG (only from the load balancer, not internet) Outbound: 5432 to DB SG (PostgreSQL) 6379 to Cache SG (Redis) 443 to 0.0.0.0/0 (for calling external APIs, via NAT Gateway) Database Security Group: Inbound: 5432 from App SG (only from app tier) Outbound: None needed (databases do not initiate connections) The security groups reference each other by group ID, not by IP. This means even if you add more app tier instances or scale down, you never have to update the database security group - it always allows the App Tier SG. Why two AZs for every tier: Every subnet is duplicated across two availability zones. If AWS AZ-a has an outage (happens rarely but does happen), your ALB automatically routes to AZ-b, the app tier has instances in AZ-b, and the RDS standby in AZ-b automatically promotes to primary. Your service continues with a brief interruption. What to mention about the NAT Gateway: NAT Gateway allows instances in private subnets to make outbound internet connections (to download packages, call external APIs) without being reachable from the internet. It sits in the public subnet and routes outbound traffic for private instances. NAT Gateway is priced by data transfer - at high volume it becomes expensive. For EC2 instances downloading S3 data, use a VPC Endpoint for S3 instead (free and keeps traffic inside AWS). ---
Tier 2 - Real Interview Questions (Q19-Q26)
19. What is the difference between AWS IAM roles and IAM users? Why should your applications never use IAM users? What the interviewer is testing: IAM security best practices. Misuse of IAM users is one of the most common AWS security vulnerabilities. Answer: IAM Users have long-lived credentials - an access key ID and secret access key that do not expire until you explicitly rotate or delete them. They are designed for human console access or legacy use cases. IAM Roles are assumable identities with temporary credentials. When a service assumes a role, it gets a temporary access key, secret key, and session token that expire after 1-12 hours (configurable). The STS (Security Token Service) issues these automatically. Why applications should never use IAM users: Long-lived credentials are dangerous for three reasons: 1. They can be stolen and used indefinitely. An access key committed to Git, leaked in logs, or copied to a developer's laptop remains valid until someone manually rotates it. This is the source of the vast majority of AWS account compromises. 2. Rotation is operationally painful. Rotating an access key requires updating it everywhere it is used - CI environment variables, application config, servers. Teams avoid doing it. Keys end up being years old. 3. No automatic expiry. Even if a key is stolen, the attacker has unlimited time to use it. Temporary credentials from STS expire - even if stolen, they become useless within hours. The alternatives: For EC2 instances: IAM Instance Role - credentials are available via the instance metadata service and rotate automatically. For EKS pods: IRSA (IAM Roles for Service Accounts) - assigns an IAM role to a Kubernetes service account. Pods using that service account get temporary credentials via the AWS SDK automatically. yaml Annotate the Kubernetes service account with the IAM role ARN apiVersion: v1 kind: ServiceAccount metadata: name: payment-service annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payment-service-role For CI/CD pipelines: OIDC federation - GitHub Actions or GitLab CI assume an IAM role directly using an OIDC token. No static credentials stored anywhere. For Lambda: Execution role - Lambda assumes the role automatically when the function runs. The only legitimate use for IAM users: Human console access combined with MFA, or legacy tooling that genuinely cannot use roles. In all other cases, roles are the correct answer. --- Incident Management and On-Call 20. You are on-call. At 11 PM you get paged - the checkout service error rate has jumped from 0.1% to 8%. Walk me through your complete response. What the interviewer is testing: On-call maturity and incident response process. Companies want to see that you are systematic, that you communicate, and that you do not make things worse by making changes under pressure. Answer: 0-2 minutes: Acknowledge and communicate Acknowledge the alert immediately. Post in the incident channel: > "Acknowledged checkout service error rate spike - 0.1% to 8%. Starting investigation. Do not push any deployments until further notice." Code freeze is the first action. The worst time for a deploy is during an active incident. 2-5 minutes: Establish a timeline bash When did this start? Check when the metric crossed the threshold Look at Grafana - was there a deployment, a traffic spike, or an external event? Recent deployments kubectl rollout history deployment/checkout-service -n production Recent events kubectl get events -n production --sort-by='.lastTimestamp' | tail -30 If a deployment happened in the last 30 minutes before the incident, that is your primary suspect. 5-10 minutes: Identify the error class bash What kind of errors? 5xx? Timeouts? In Prometheus: rate(http_requests_total{service="checkout",status=~"5.."}[5m]) by (status) Is it 500 (application error) or 504 (timeout to downstream)? Check the logs kubectl logs -l app=checkout-service -n production --tail=100 | grep "ERROR\|FATAL\|panic" Previous logs if pods have restarted kubectl logs -l app=checkout-service -n production --previous --tail=50 10-15 minutes: Determine if you need to rollback If the incident started immediately after a deployment and errors are clearly application-level: bash Rollback to previous known-good version kubectl rollout undo deployment/checkout-service -n production Watch the rollback kubectl rollout status deployment/checkout-service -n production If no recent deployment, check the downstream services - is the payment gateway timing out? Is the database connection pool exhausted? bash Is the database reachable from the app? kubectl exec -it $(kubectl get pods -l app=checkout-service -n production -o jsonpath='{.items[0].metadata.name}') \ -- nc -zv db.internal 5432 Connection pool status (application-specific) kubectl logs -l app=checkout-service | grep "pool\|connection" | tail -20 Throughout: keep stakeholders updated every 10 minutes > "Update at 23:15 - traced to database connection pool exhaustion. Rolling back last deployment which changed connection pool config. ETA 5 minutes." After resolution: write the incident timeline while it is fresh * When alert fired * When you acknowledged * Timeline of what you checked and found * What the root cause was * What you changed to fix it This becomes the post-mortem document reviewed on Monday. What makes this answer stand out in an interview: Mentioning the code freeze first, the communication cadence throughout, and writing the timeline immediately after resolution - not just the technical debugging steps. --- Advanced Topics 21. What is a service mesh and when does adding one make sense? What are the costs? What the interviewer is testing: Architecture judgment on complex infrastructure decisions. Adding a service mesh too early is a common over-engineering mistake. Not knowing when you need one is also a problem. Answer: A service mesh is an infrastructure layer that handles communication between services - traffic management, mTLS, observability, retries, circuit breaking - without requiring application code changes. Istio and Linkerd are the two most common options. It works by injecting a sidecar proxy (Envoy in Istio's case) into every pod. All traffic in and out of each pod flows through this proxy. The control plane configures all proxies centrally. Service A -> Envoy sidecar -> network -> Envoy sidecar -> Service B ↕ ↕ Istio control plane (issues certs, sets routing rules) What it gives you: * mTLS between all services with automatic certificate rotation - zero application code changes required * Distributed tracing via automatic span creation at every service boundary * Traffic splitting - send 10% of traffic to a new version while 90% goes to stable (canary at the mesh level, not the Kubernetes replica level) * Circuit breaking - stop sending traffic to a service that is failing repeatedly * Retries and timeouts - configurable per route without changing application code * Detailed per-service metrics automatically When it makes sense: * You have 20+ services and the operational complexity of managing security policies, retries, and observability per-service is becoming unsustainable * You have a compliance requirement that all inter-service traffic must be encrypted and authenticated * You want traffic management capabilities (canary, A/B testing) at a granularity that Kubernetes Services cannot provide The real costs - and why you should not add one prematurely: CPU and memory overhead. Every pod now runs two containers - your app and the Envoy sidecar. Envoy uses 50-200MB of memory per pod and adds measurable CPU overhead to every request. For a 100-pod cluster, you are paying for 100 extra Envoy instances. Debugging complexity. When something breaks, you now have to determine: is this a failure in my application, or in the Envoy proxy, or in the Istio control plane? The blast radius of a misconfigured mesh policy is every service in the mesh simultaneously. Operational overhead. Istio upgrades are non-trivial. The CRD surface area is large. Debugging network policies requires understanding of DestinationRules, VirtualServices, and PeerAuthentication objects. The honest answer for an interview: "For a team running 10 services, I would add mTLS with cert-manager and external-dns before reaching for Istio. The value of a full service mesh comes at 30+ services where the per-service configuration maintenance cost is genuinely higher than the mesh overhead." --- 22. You are asked to improve your organization's deployment process to reduce the mean time to recover (MTTR) from failed deployments. What changes do you make? What the interviewer is testing: Reliability engineering thinking - not just deploying faster but recovering faster. Answer: MTTR is determined by three things: time to detect a problem, time to decide to roll back, and time to execute the rollback. Every improvement targets one of these. Reduce time to detect - better alerting and deployment monitoring Most teams find out about a bad deployment from users, not monitoring. Fix this first. Add a deployment verification step that runs immediately after every production deploy: bash Smoke tests that run automatically after deployment These should complete in under 2 minutes and test the critical user paths Is the service returning 200? curl -sf https://api.company.com/health | jq '.status == "ok"' Is the critical endpoint working? curl -sf -X POST https://api.company.com/checkout \ -H "Content-Type: application/json" \ -d '{"test": true}' | jq '.success == true' Did error rate spike vs baseline? Check Prometheus alert within 2 minutes of deployment Add a deployment marker to Grafana. When you deploy, draw a vertical line on every dashboard at that timestamp. The moment error rate changes, the correlation to the deployment is visually immediate. Reduce time to decide - auto-rollback with clear criteria Manual rollback decisions waste time. Define the criteria upfront: yaml ArgoCD progressive delivery with automatic rollback Using Argo Rollouts spec: strategy: canary: analysis: templates: - templateName: error-rate-check args: - name: service-name value: checkout-service ## Automatically rollback if analysis fails autoPromotionEnabled: false The rule: if error rate or latency SLO is breached within 5 minutes of a deployment, automatically roll back without waiting for human decision. Reduce time to execute - one-command rollback bash Kubernetes rolling update - rollback is instant kubectl rollout undo deployment/checkout-service -n production This takes 30-60 seconds Make sure this is documented, tested regularly, and every on-call engineer knows it Not knowing the rollback command at 2 AM costs you 5-10 minutes The process change that matters most: Run a quarterly "game day" where you deliberately deploy a bad version and practice detecting and rolling back. Most teams discover in these exercises that the rollback command is not documented, the smoke tests do not run automatically, or the deployment marker was never added to the dashboard. Discover this in a game day, not in a production incident. --- 23. What is the difference between a liveness probe and a readiness probe in Kubernetes? What happens if you configure them wrong? What the interviewer is testing: Most engineers know what these probes are. Fewer understand the failure consequences - which is where production incidents come from. Answer: Both probes run on a schedule and check whether a container is healthy. The difference is what Kubernetes does when they fail. Readiness probe - if this fails, Kubernetes removes the pod from the Service endpoints. Traffic stops routing to it. The pod keeps running but receives no requests. When the probe passes again, it gets added back. Use this for: application startup (pod is running but not ready yet), temporary overload, warming up a cache. Liveness probe - if this fails, Kubernetes kills and restarts the container. Use this for: detecting deadlocks or stuck states where the application process is alive but not making progress. yaml livenessProbe: httpGet: path: /health/live ## must return 200 if the process is functional port: 8080 initialDelaySeconds: 30 ## wait 30s after container starts before first check periodSeconds: 10 ## check every 10s failureThreshold: 3 ## restart after 3 consecutive failures readinessProbe: httpGet: path: /health/ready ## return 200 only when ready to serve traffic port: 8080 initialDelaySeconds: 5 periodSeconds: 5 failureThreshold: 3 What breaks when configured wrong: Liveness probe too aggressive (low initialDelaySeconds): The app takes 45 seconds to start. Liveness probe starts at 10 seconds, sees a failure, restarts the container. The container never finishes starting. This is a CrashLoopBackOff that is entirely caused by the probe - the app code is fine. Liveness probe pointing at a dependency: Your liveness probe calls /health which checks database connectivity. Your database goes down briefly. Every pod in the deployment gets restarted simultaneously because they all fail the liveness check. You now have zero running pods during a database blip that would have been survivable if the pods had stayed up. No readiness probe at all: Kubernetes routes traffic to new pods immediately after the container starts - before the application has finished loading. The first real requests hit an app that is not ready. Users see errors during every deployment. > 📌 Remember: Liveness and readiness should check different things. Liveness checks "is this process functional." Readiness checks "is this process ready to serve traffic." Your liveness probe should never check external dependencies - only the health of the local process. --- 24. How does Kubernetes handle secret storage and what are its security limitations? What is the better alternative? What the interviewer is testing: Security maturity around secrets management. This is a mid-level filter question at security-conscious companies. Answer: Kubernetes Secrets store data as base64-encoded strings in etcd. They are not encrypted by default - base64 is encoding, not encryption. Anyone with access to etcd (or a backup of etcd) can read every Secret in plain text. bash A Secret that looks protected: kubectl get secret db-credentials -o yaml apiVersion: v1 kind: Secret data: password: cGFzc3dvcmQxMjM= <- this is just base64 Decode it trivially: echo "cGFzc3dvcmQxMjM=" | base64 --decode password123 The limitations of Kubernetes Secrets: * No rotation - changing a secret requires manually updating the Secret object and restarting pods * No audit trail - you cannot easily see who read a secret or when * No expiry - secrets do not expire automatically * etcd encryption at rest is opt-in and not enabled by default in most clusters * Any pod with the right RBAC permissions can read any Secret in its namespace The better alternative - external secrets managers: Use AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager as the source of truth. Sync them into Kubernetes using External Secrets Operator (ESO): yaml ExternalSecret object - fetches from AWS Secrets Manager apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-credentials spec: refreshInterval: 1h ## re-sync every hour (picks up rotations) secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore target: name: db-credentials ## creates a Kubernetes Secret with this name data: - secretKey: password remoteRef: key: prod/db/credentials property: password The actual secret value lives in AWS Secrets Manager where you have: encryption at rest (KMS), audit logs in CloudTrail, automatic rotation, and fine-grained IAM access control. The Kubernetes Secret is just a synchronized copy. What to say at the end: "I would also enable envelope encryption for etcd in the cluster config, even if using external secrets, because it is a cheap additional layer for any secrets that do land in etcd." --- 25. What is a Helm chart and when does Helm make your life harder instead of easier? What the interviewer is testing: Tool judgment. Over-reliance on Helm is a common mid-level anti-pattern. Answer: Helm is a package manager for Kubernetes. A Helm chart is a collection of Kubernetes manifests with Go templating - values can be substituted at deploy time, making the same chart work for dev, staging, and production with different configs. bash Deploy with environment-specific values helm upgrade --install payment-service ./charts/payment \ --namespace production \ --values values-production.yaml \ --set image.tag=v2.3.1 Rollback to previous release helm rollback payment-service 1 When Helm is the right choice: * Deploying third-party software (databases, monitoring tools, ingress controllers) where the chart already exists and is maintained * Applications with genuinely different configurations across 3+ environments where managing separate manifests causes real duplication * Teams where templating and packaging standards improve consistency across multiple services When Helm makes things harder: Debugging is harder. When helm upgrade fails, the error often points into the rendered template, not your values file. You have to run helm template to see what was actually generated, then find where in your values the problem originated. bash Render the chart locally to debug what Helm actually generates helm template payment-service ./charts/payment \ --values values-production.yaml \ > rendered.yaml Now you can read and validate the actual YAML that would be applied Chart drift. If you install a Helm chart and then someone does kubectl edit on a resource Helm manages, the next helm upgrade will overwrite that manual change silently. Helm owns what it deploys - manual edits outside Helm break this contract. Simple services do not need it. A single-service deployment with a Deployment, Service, and ConfigMap - three files totalling 80 lines - does not benefit from Helm templating. Adding Helm adds a values.yaml, a Chart.yaml, a templates directory, and dependency on the Helm binary. The honest mid-level answer: use Helm for shared infrastructure and third-party charts. For your own services, evaluate whether the templating complexity is actually paying for itself. --- 26. What is container image scanning and where in your pipeline does it belong? What the interviewer is testing: DevSecOps awareness. Security shifted left into the pipeline is expected at mid-level. Answer: Container image scanning checks your Docker image against a database of known vulnerabilities (CVEs) in the OS packages, language runtimes, and libraries your image contains. Tools: Trivy (most common), Snyk, Grype, AWS ECR native scanning. Where it belongs - and why order matters: Build image -> Scan image -> Push to registry -> Deploy ^ Block here if CRITICAL CVEs found Scan before push. If you scan after push or after deploy, vulnerable images are already in your registry or running in production. The entire point is to catch issues before they are deployed. yaml GitHub Actions example - name: Build image run: docker build -t myapp:${{ github.sha }} . - name: Scan image for vulnerabilities run: | trivy image \ --exit-code 1 \ ## fail the pipeline on findings --severity CRITICAL,HIGH \ ## block on CRITICAL and HIGH --ignore-unfixed \ ## skip CVEs with no fix available (noise reduction) myapp:${{ github.sha }} - name: Push to registry if: success() ## only push if scan passed run: docker push myapp:${{ github.sha }} What to configure to avoid false positives: * --ignore-unfixed skips CVEs that have no patch available yet - you cannot fix them, they just create noise * Set severity threshold to CRITICAL only for strict pipelines, CRITICAL,HIGH for balanced * Create a .trivyignore file for known false positives with justification comments Beyond the pipeline - continuous scanning: The pipeline scan catches vulnerabilities at build time. New CVEs are published every day against images you built months ago. Add continuous registry scanning: ECR has this built in, or use Trivy Operator in Kubernetes which scans running pods and reports findings as Kubernetes resources. bash Check Trivy Operator findings in your cluster kubectl get vulnerabilityreports --all-namespaces ---
Tier 2 - Real Interview Questions (Q27-Q30)
27. What is a circuit breaker pattern and how would you implement it for a service that calls an unreliable external API? What the interviewer is testing: Resilience patterns. This is an architecture question that shows whether you think about failure modes, not just happy paths. Answer: A circuit breaker prevents your service from continuously hammering a failing downstream dependency. Without it, if the payment gateway is down, every request to your checkout service waits the full timeout period (say 30 seconds) before failing. Under load, you exhaust your thread pool waiting on a dependency that is not responding. Your checkout service goes down because the payment gateway is down - even if checkout could have degraded gracefully. The circuit breaker has three states: CLOSED (normal) -> requests pass through to the external API | | failure rate exceeds threshold (e.g. 50% failures in 10s window) v OPEN (tripped) -> requests immediately return an error without calling the API | | after a timeout (e.g. 30 seconds), allow one test request v HALF-OPEN (testing) -> if test request succeeds -> back to CLOSED -> if test request fails -> back to OPEN Implementation with resilience4j (Java) or the pattern in Python: python import time from enum import Enum class CircuitState(Enum): CLOSED = "closed" OPEN = "open" HALF_OPEN = "half_open" class CircuitBreaker: """ Wraps calls to an external service. Opens after failure_threshold consecutive failures. Retests after recovery_timeout seconds. """ def __init__(self, failure_threshold=5, recovery_timeout=30): self.state = CircuitState.CLOSED self.failure_count = 0 self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.last_failure_time = None def call(self, func, args, *kwargs): if self.state == CircuitState.OPEN: ## Check if recovery timeout has elapsed if time.time() - self.last_failure_time > self.recovery_timeout: self.state = CircuitState.HALF_OPEN else: raise Exception("Circuit breaker OPEN - fast fail, not calling external API") try: result = func(args, *kwargs) self._on_success() return result except Exception as e: self._on_failure() raise e def _on_success(self): self.failure_count = 0 self.state = CircuitState.CLOSED def _on_failure(self): self.failure_count += 1 self.last_failure_time = time.time() if self.failure_count >= self.failure_threshold: self.state = CircuitState.OPEN In production with a service mesh: Istio implements circuit breaking at the proxy level with no application code changes: yaml apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: payment-gateway spec: host: payment-gateway.external.com trafficPolicy: outlierDetection: consecutive5xxErrors: 5 ## trip after 5 consecutive 5xx responses interval: 10s ## evaluated over a 10s window baseEjectionTime: 30s ## eject the host for 30s What to say in the interview: "The circuit breaker is paired with a fallback - what do you do when the circuit is open? For a payment service the answer might be 'queue the request for retry.' For a recommendation service it might be 'return cached results.' Having no fallback means the circuit breaker just moves the error, not resolves it." --- 28. How do you handle rollbacks in a GitOps workflow where the deployment is triggered by a Git commit? What the interviewer is testing: GitOps operational depth. Many engineers understand the happy path but not the rollback story. Answer: In a GitOps workflow, Git is the source of truth. The correct rollback is a Git operation - not a direct kubectl rollout undo. Running kubectl rollout undo outside of Git means your live cluster state diverges from what Git says it should be. ArgoCD will detect the drift and reconcile it back - undoing your manual rollback. The correct rollback flow: bash Option 1 - Revert the commit that triggered the bad deployment git log --oneline a3f8d21 update image tag to v2.4.0 <- bad deployment b1c9e44 update image tag to v2.3.9 <- last known good git revert a3f8d21 ## creates a new commit that undoes the image tag change git push origin main ## ArgoCD detects the change and deploys v2.3.9 bash Option 2 - If you use a separate config/gitops repo (common pattern) CI writes image tag to the gitops repo Rollback means reverting that commit in the gitops repo cd gitops-repo git log --oneline deployments/payment-service/ Revert the last tag update git revert HEAD git push ArgoCD syncs within 3 minutes (or immediately if you trigger it) Speed matters - ArgoCD manual sync: Auto-sync has a polling interval (usually 3 minutes). If the incident is severe and you cannot wait 3 minutes: bash Force an immediate sync after your git revert argocd app sync payment-service Deployment starts within seconds The pattern that makes rollbacks fast - separate image tag from everything else: Keep your application code repo and your Kubernetes manifest repo separate. The CI pipeline only changes one line - the image tag - in the manifests repo. A rollback is reverting one line in one file. This is fast, auditable, and safe. application-repo/ <- developer code, CI builds image here gitops-repo/ deployments/ payment-service/ deployment.yaml <- CI auto-updates image.tag here only What to say in the interview: "The key principle is: you never bypass Git in a GitOps setup. Even in an incident. If you run kubectl rollout undo manually, your next ArgoCD sync will re-apply the broken version from Git and undo your fix. The fix always goes through Git first." --- 29. What is the difference between horizontal and vertical scaling for a Kubernetes cluster itself - adding nodes versus resizing nodes? When does each make sense? What the interviewer is testing: Cloud infrastructure cost and scaling judgment at the cluster level, not just pod level. Answer: Horizontal cluster scaling (add more nodes): Add worker nodes to the cluster. Each new node brings its own CPU, memory, and pod capacity. This is what Cluster Autoscaler does - watches for pending pods that cannot schedule due to insufficient resources, and adds nodes to accommodate them. bash Cluster Autoscaler adds nodes automatically when pods are Pending kubectl get pods -n kube-system | grep cluster-autoscaler Check current node count kubectl get nodes Vertical cluster scaling (resize existing nodes): Replace existing nodes with larger instance types - upgrading from t3.xlarge (4 vCPU, 16GB) to c5.2xlarge (8 vCPU, 16GB). In AWS this means creating a new node group with the larger instance type and draining the old nodes into it. When to choose horizontal: * Your workload is stateless and scales by adding replicas * You want fault tolerance - more nodes means one node failure has smaller impact * Your pods have moderate resource needs and many of them fit on standard nodes * You need to scale up and down quickly - Cluster Autoscaler can add nodes in 2-3 minutes When to choose vertical (larger nodes): * You have pods with large memory requirements that do not fit on standard nodes - ML model serving, JVM-based services with large heap sizes * You want to reduce node count for easier management (fewer nodes to patch, drain, monitor) * Your workload has high pod-to-pod communication - co-locating more pods on fewer, larger nodes reduces network hops * Spot/preemptible node costs make larger nodes more cost-effective per vCPU The production answer - use both: Most mature clusters use multiple node groups: a standard node group for general workloads scaled horizontally, and a high-memory or GPU node group for specialized workloads. Kubernetes node selectors and taints route workloads to the appropriate node type. yaml Node selector - this pod only schedules on memory-optimized nodes spec: nodeSelector: node.kubernetes.io/instance-type: r5.2xlarge --- 30. Your organization is moving from a monolithic deployment to microservices. You are asked to design the CI/CD strategy for 20 new services. What decisions do you make upfront and why? What the interviewer is testing: Architecture-level thinking about CI/CD at scale. This is a design question - there is no single correct answer, but weak answers ignore the tradeoffs entirely. Answer: Getting CI/CD architecture right for 20 services from the start is far cheaper than refactoring 20 independent pipelines later. The decisions that matter most upfront: Decision 1 - Monorepo or polyrepo Monorepo (all 20 services in one Git repo) gives you: atomic commits across services, shared tooling and conventions, easier dependency management between services. The cost: CI must be smart about path-based builds or every commit rebuilds all 20 services. Polyrepo (each service in its own repo) gives you: clear ownership, independent versioning, no accidental coupling. The cost: cross-service changes require multiple PRs, shared tooling must be distributed, harder to enforce standards. For a new migration, monorepo with path-based CI is usually the better starting point - you can always split later, but merging is painful. Decision 2 - Shared pipeline templates Write one pipeline template that all 20 services reuse. Individual services override only what is different. yaml GitHub Actions reusable workflow - one definition, 20 callers .github/workflows/service-pipeline.yml (template) on: workflow_call: inputs: service_name: required: true type: string dockerfile_path: default: "./Dockerfile" type: string Each service calls it with 3 lines: uses: ./.github/workflows/service-pipeline.yml with: service_name: payment-service If each of 20 services has its own independent pipeline YAML, you have 20 places to fix when a security scanner version needs updating. With templates, you fix one file. Decision 3 - Shared image registry with consistent tagging All services push to the same ECR or Docker Hub org. Every image tagged with git-sha, branch-name, and latest on main. This makes traceability trivial - which commit is running in production is always answerable. Decision 4 - Deployment via GitOps from day one Start with a GitOps deployment model (ArgoCD) before you have 20 services, not after. Retrofitting GitOps onto 20 independent deploy scripts is a multi-month project. Starting with it means all 20 services get consistent deployment, rollback, and drift detection automatically. Decision 5 - Shared observability standards Every service must expose /health/live, /health/ready, and /metrics before it gets a pipeline. This is not optional - it is the entry ticket to the CI/CD system. Services that do not instrument themselves do not deploy to production. What you say in the interview: "The most expensive CI/CD mistake is treating each service as a unique snowflake. The value of microservices is independent deployability, not independent pipelines. Shared templates and standards are what make 20 services manageable by a 3-person platform team." ---
Skills You'll Master
Curriculum Index
Before You Read This
You cleared the Junior round. You can explain what a pod is, you know how Docker layers work, and you can write a basic ...
Tier 1 - Mid-Level Fundamentals Checklist
> If you need to look any of these up, go back to the relevant module first. These are the floor, not the ceiling. Kuber...
Tier 2 - Real Interview Questions (Q1-Q10)
CI/CD - Architecture and Decisions 1. Your team has a monorepo with 12 microservices. The CI pipeline builds and tests a...
Tier 2 - Real Interview Questions (Q11-Q18)
11. You are designing an observability stack for a new microservices deployment with 30 services. Walk me through your c...
Tier 2 - Real Interview Questions (Q19-Q26)
19. What is the difference between AWS IAM roles and IAM users? Why should your applications never use IAM users? What t...
Tier 2 - Real Interview Questions (Q27-Q30)
27. What is a circuit breaker pattern and how would you implement it for a service that calls an unreliable external API...
Tier 3 - Scenario Round
31. It is Thursday afternoon. Your team is about to do a large deployment - a rewrite of the authentication service that...
Behavioral Round
41. Tell me about a time you introduced a change that caused a production incident. What happened, and what did you chan...
Salary Reference
Mid-Level DevOps Engineer (2-5 years) Company Type Range Service / IT company Rs 10L - Rs 18L Mid-size product startup R...
Career Impact
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.
Troubleshooting & Lab Guides
Frequently Asked Questions
Yes - every question here reflects the shape and depth of what mid-level candidates are actually asked at product companies, not textbook trivia. Answers are written from scratch, not copied from any source.
This pack targets engineers with roughly 2-5 years of experience moving from Junior to Mid-level DevOps, SRE, or Platform Engineer roles.
Budget around 2 hours to read through all four tiers properly, longer if you work through the scenario round out loud or with a study partner.
No. Focus on understanding the reasoning behind each answer - interviewers are testing whether you understand the tradeoffs, not whether you can recite a script.