- 2-5 years experience - Roles: DevOps Engineer, Site Reliability Engineer, Platform Engineer - 70 checklist questions, 30 real interview Q&A, 10 live scenarios, 10 behavioral questions - Companies: Atlassian, Stripe, PhonePe, Cloudflare, Grab, Databricks
You cleared the Junior round. You can explain what a pod is, you know how Docker layers work, and you can write a basic Jenkins pipeline. That is table stakes here — nobody at a mid-level interview will ask you to define Kubernetes. What changes at mid-level is what you are expected to own. Junior engineers execute. Mid-level engineers design, troubleshoot systems they did not build, make tradeoff decisions, and explain why they made them. The questions shift from "what is X" to "you have a production problem, walk me through it" and "your team wants to do X, how do you architect it and what breaks?" This module has four tiers. Every question across all tiers is numbered continuously so you always know exactly where you are. **Tier 1 — Fundamentals Checklist (no answers)** 70 questions across Kubernetes, CI/CD, IaC, Observability, Networking/Security, Docker, GitOps, and Incident Management. These have no answers written out. If you cannot answer any of them cold, go back to the relevant module first. These are the floor — the minimum expected of every mid-level candidate before the interview starts. **Tier 2 — Real Interview Questions (Q1 to Q30)** 30 questions at mid-level depth with full answers — the kind asked at companies like Atlassian, Cloudflare, Stripe, PhonePe, Grab, and well-funded product startups. Every answer explains what the interviewer is actually testing and what separates a good answer from a mediocre one. Topics: CI/CD architecture, Kubernetes operations, SRE and observability, Terraform, security, Docker, and AWS. **Tier 3 — Scenario Round (Q31 to Q40)** 10 open-ended production scenarios. No single correct answer exists — the interviewer is watching how you think, prioritize, and communicate under pressure. These replicate the live whiteboard or screen-share rounds used at most mid-market and enterprise companies. Silence is what eliminates candidates here, not wrong answers. **Tier 4 — Behavioral Round (Q41 to Q50)** 10 behavioral questions with guidance on what a strong answer looks like. Mid-level behavioral rounds test ownership, tradeoff thinking, mistake handling, and cross-team communication — not just technical execution. Every question here has appeared in real mid-level loops. ---
> If you need to look any of these up, go back to the relevant module first. These are the floor, not the ceiling. **Kubernetes — operational depth** * What is the difference between a Deployment, StatefulSet, and DaemonSet? When do you use each? * What is a PodDisruptionBudget and when does it matter? * How does the Kubernetes scheduler decide which node to place a pod on? * What is the difference between a ClusterIP, NodePort, and LoadBalancer service? * How do you configure resource requests and limits, and what happens when a container exceeds its memory limit? * What is a HorizontalPodAutoscaler and what metrics can it scale on? * What is the difference between a ConfigMap and a Secret? * How does RBAC work in Kubernetes? What is the difference between a Role and a ClusterRole? * What is an Ingress and how does it differ from a Service? * What are taints and tolerations used for? * What is the difference between a liveness probe and a readiness probe? * What happens to a pod when its node is drained? * How does Kubernetes handle rolling updates — what do maxSurge and maxUnavailable control? * What is an init container and when would you use one? * What is the purpose of a NetworkPolicy in Kubernetes? **CI/CD — architecture level** * What is the difference between a pipeline trigger on push vs on pull request? * How do you handle database migrations in a CI/CD pipeline? * What is a monorepo and what challenges does it create for CI/CD? * How do you prevent secrets from leaking in CI logs? * What is a pipeline artifact and how do you share it between stages? * What is the difference between a canary deployment and a blue/green deployment? * How do you implement an automatic rollback in a CI/CD pipeline? * What is a deployment gate and when do you use one? * How do you version Docker images in a pipeline — what tagging strategy do you use and why? * What is the difference between a static code analysis tool and a DAST tool in a pipeline? * What is dependency caching in CI and how does it work? **Infrastructure as Code** * What is Terraform state drift and how do you detect it? * What is the difference between `terraform destroy` and `terraform apply` with a removed resource? * How do Terraform modules work and why do you use them? * What is the difference between count and for_each in Terraform? * What is a Terraform workspace? * What is the difference between `terraform import` and writing a resource from scratch? * What is a data source in Terraform and when do you use it instead of a resource? * How do you handle sensitive values like passwords in Terraform without committing them to Git? **Observability** * What is the difference between a metric, a log, and a trace? * What is an SLO and an error budget? * What is cardinality in Prometheus and why does high cardinality cause problems? * What is a histogram metric and when is it more useful than a gauge? * What is the difference between a counter and a gauge in Prometheus? * What is AlertManager and what does it do that Prometheus alone cannot? * What is a recording rule in Prometheus and when do you write one? * What is OpenTelemetry and why does it matter for tracing? **Networking and Security** * What is mTLS and why is it used in microservices? * What is a service mesh and what problem does it solve? * What is the difference between authentication and authorization? * What is OIDC and how does it work with Kubernetes? * How does a VPN differ from a bastion host for accessing private infrastructure? * What is the difference between a Security Group and a NACL in AWS? * What is a VPC endpoint and when do you use one instead of routing traffic through NAT Gateway? * What is the principle of least privilege and how do you apply it to IAM roles? * What is IRSA (IAM Roles for Service Accounts) and why is it better than using node-level IAM roles? **Docker and Containers** * What is the difference between a Docker image layer and a container layer? * What is a multi-stage Docker build and what problem does it solve? * What does `.dockerignore` do and why does it matter for build performance? * What is the difference between `CMD` and `ENTRYPOINT` in a Dockerfile? * What happens to data written inside a container when the container is removed? * What is the difference between a bind mount and a named volume in Docker? * How do you reduce Docker image size — name three concrete techniques? **GitOps** * What is the difference between push-based and pull-based deployment? * How does ArgoCD know when a deployment is out of sync? * What is Flux and how does it differ from ArgoCD? * What is a GitOps reconciliation loop and what triggers it? * How do you handle secrets in a GitOps workflow where everything is committed to Git? **Incident Management and Reliability** * What is MTTR and MTBF? Which one matters more for user experience? * What is a postmortem and what makes one blameless? * What is the difference between a P1 and P2 incident — how does your response differ? * What is a runbook and how does it differ from a playbook? * What is chaos engineering and what problem does it solve? * What is alert fatigue and what causes it? * How do you decide what severity level to assign an alert? ---
### CI/CD — Architecture and Decisions ### 1. Your team has a monorepo with 12 microservices. The CI pipeline builds and tests all 12 on every commit. It takes 40 minutes. How do you fix this? **What the interviewer is testing:** Can you solve a real pipeline architecture problem — not just say "make it faster." This is one of the most common senior/mid-level screen questions. **Answer:** The first thing to say in an interview: "Before changing anything, I would measure where the time is actually going." Guessing wastes engineering time. ```bash ## Look at stage-by-stage timing in your CI provider (GitHub Actions, GitLab CI) ## Most CI tools show per-step duration in the pipeline run logs ## Typically one of three things is responsible for 70%+ of the time: ## 1. Dependency installation (npm ci, pip install, go mod download) ## 2. Test execution (usually the biggest) ## 3. Docker image builds ``` Once you know the bottleneck, the fix depends on what it is. For a monorepo specifically, the architectural solution is **path-based change detection** — only build and test the services that actually changed. Git diff between current commit and last successful build | v Identify which directories changed | v Map directories to service names | v Only trigger CI jobs for affected services In GitHub Actions this looks like: ```yaml ## Use paths-filter to detect which services changed - uses: dorny/paths-filter@v2 id: changes with: filters: | payments: - 'services/payments/**' orders: - 'services/orders/**' ## Only run payments tests if payments code changed - name: Test payments service if: steps.changes.outputs.payments == 'true' run: cd services/payments && npm test ``` Beyond path detection, add these in order of impact: **Cache dependencies aggressively.** If `package-lock.json` or `go.sum` has not changed, restore the cache instead of downloading again. This is often a 3-5 minute saving per service. **Parallelize across services.** Run the CI jobs for affected services simultaneously, not sequentially. Most CI systems support matrix builds or parallel job graphs. **Separate fast gates from slow gates.** Lint and unit tests should run on every push and complete in under 3 minutes. Integration tests, end-to-end tests, and security scans should only run on PRs to main. **Use Docker layer caching.** Structure your Dockerfile so the dependency installation layer is cached separately from the source code layer. The dependency layer only invalidates when the lock file changes. ```dockerfile ## Copy lock files first — this layer caches as long as deps are unchanged COPY package.json package-lock.json ./ RUN npm ci ## Copy source code after — this layer rebuilds on every code change COPY src/ ./src/ RUN npm run build ``` Done well, a 40-minute pipeline for a 12-service monorepo can come down to 6-8 minutes for a typical PR that touches 1-2 services. **What to say at the end:** "I would also talk to the developers about what they actually need fast feedback on. Often teams are running expensive integration tests on every push because nobody questioned the default. Fast feedback on a subset is more valuable than slow feedback on everything." --- ### 2. You need to implement a CI/CD pipeline that handles database schema migrations safely. What does your pipeline look like? **What the interviewer is testing:** Database migrations are where most deployment strategies break. This question separates engineers who have done it from those who have only read about it. **Answer:** The core problem is that migrations and application deployments are two separate things that happen to need coordination. Getting this wrong causes either downtime or data loss. The general rule: **migrations must be backward compatible with both the old and new version of the application code.** This means you cannot add a NOT NULL column without a default in the same deploy as the code that writes to it — because during the rollout window, old pods are still running and they do not write that column. A safe migration pipeline looks like this: Stage 1: Run migration in a transaction | v Stage 2: Verify migration succeeded (check schema version) | v Stage 3: Deploy new application version | v Stage 4: Run smoke tests | v Stage 5 (optional): Run cleanup migration to remove old columns Breaking a migration across multiple deploys: **Deploy 1 — additive only:** ```sql -- Safe: add column with a default, nullable, or backfilled ALTER TABLE orders ADD COLUMN status_v2 VARCHAR(50) DEFAULT 'pending'; ``` New application code writes to both `status` and `status_v2`. Old code still writes only `status`. Nothing breaks. **Deploy 2 — switch reads:** Application now reads from `status_v2`. Old code reading from `status` has been fully replaced. **Deploy 3 — cleanup:** ```sql -- Safe to drop old column now that no code references it ALTER TABLE orders DROP COLUMN status; ``` **In the pipeline itself:** ```yaml ## Migration stage runs before app deployment - name: Run database migration run: | ## Use a migration tool with version tracking (Flyway, Liquibase, golang-migrate) flyway migrate -url=$DATABASE_URL ## Fail the pipeline if migration fails — never deploy an app to a broken schema - name: Verify migration version run: | flyway info -url=$DATABASE_URL | grep "Success" ``` > ⚠️ **Security:** Never put the database connection string directly in pipeline YAML. Pull it from a secrets manager at runtime. **The question interviewers follow up with:** "What do you do if a migration runs successfully but the app deployment fails?" Answer: Your migration is committed. The database is in the new schema. Rolling back the app is safe only if the migration was backward compatible — which it should be by design. If it was not, you now have a problem. This is exactly why the expand-then-contract pattern (three deploys above) exists. --- ### 3. What is GitOps and how is it architecturally different from a standard push-based CI/CD pipeline? **What the interviewer is testing:** GitOps is a 2026 filter question. Companies running Kubernetes at scale have almost universally moved toward it. Knowing the topology is not enough — you need to explain the security model. **Answer:** In a standard push-based pipeline, the CI system (GitHub Actions, Jenkins) has credentials to your Kubernetes cluster and deploys directly. Developer pushes code | v CI pipeline builds image, runs tests | v CI pipeline runs: kubectl apply or helm upgrade | v ← CI system needs cluster credentials Kubernetes cluster is updated In a GitOps pull-based model, the CI system has no cluster credentials at all. Instead, an operator running inside the cluster watches a Git repository and applies whatever is declared there. Developer pushes code | v CI pipeline builds image, pushes to registry | v CI pipeline updates image tag in Git (the "desired state" repo) | v ArgoCD/Flux detects Git has changed | v ← cluster reaches OUT to Git, not the other way Operator applies changes to the cluster from inside **Why this matters from a security standpoint:** In the push model, if your CI system is compromised — a malicious pipeline, a stolen token, a supply chain attack — the attacker has direct access to your production Kubernetes cluster. Every build agent has cluster admin credentials sitting in environment variables. In the pull model, no external system has cluster credentials. The cluster connects out to Git, reads what it should look like, and reconciles. An attacker who compromises your CI pipeline can push a bad image to the registry, but they cannot directly execute commands against your cluster. **ArgoCD specifically:** ArgoCD runs as a controller inside your cluster. It compares the live cluster state to what is declared in your Git repo every few minutes (configurable). If they diverge — because someone ran `kubectl apply` manually, or a deployment was edited directly — ArgoCD marks the application as `OutOfSync` and can automatically reconcile it back to what Git says. ```bash ## Check sync status argocd app get my-application ## Manually sync if auto-sync is off argocd app sync my-application ## See what changed (drift detection) argocd app diff my-application ``` > 📌 **Remember:** GitOps does not replace CI. CI still builds images and runs tests. GitOps only replaces the deployment step. The two work together — CI produces the artifact, GitOps deploys it. --- ### Kubernetes — Operational Depth ### 4. A service is responding to 97% of requests normally but 3% are timing out. Health checks are passing. How do you diagnose this? **What the interviewer is testing:** This is the most sophisticated Kubernetes debugging question at mid-level. It specifically tests whether you understand that health checks passing does not mean the service is healthy. **Answer:** The key insight here is that health checks only test if the pod is alive, not if it is working correctly for all requests. A pod with a memory leak or a deadlocked thread pool can pass a health check and still fail real traffic. **Step 1 — Isolate whether the issue is infrastructure or application** ```bash ## Check if the errors are concentrated on specific pods kubectl get pods -n production -l app=payment-service ## Note the pod names ## Check per-pod error rates in your metrics ## In Prometheus: rate(http_requests_total{status=~"5..",job="payment-service"}[5m]) by (pod) ## If one pod shows 30% errors while others show 0%, that pod is sick ``` If the errors are concentrated on one pod, that pod has a problem the health check is not catching — often a thread pool exhaustion or a broken database connection that was established before the check ran. **Step 2 — Check if it is a connection draining issue** A 3% failure rate on a 3-replica deployment strongly suggests one pod is unhealthy. The Service load balances roughly evenly across 3 pods — if one pod is rejecting 100% of its requests, you see approximately 33% failures. If it is rejecting ~10%, you see ~3% overall. This is a classic signature. ```bash ## Check if a pod is consistently slow or failing kubectl top pods -n production -l app=payment-service ## Look for a pod with noticeably higher CPU or memory ## Check logs specifically on the suspected pod kubectl logs <specific-pod-name> -n production --tail=200 | grep -i "error\|timeout\|exception" ``` **Step 3 — Check connection pool exhaustion** This is one of the most common causes. The application has a fixed pool of database connections. Under load, if connections are not returned quickly (slow queries, long transactions, connection leaks), new requests queue waiting for a connection, then time out. ```bash ## Inside the pod — check active connections to the database kubectl exec -it <pod-name> -- netstat -an | grep :5432 | grep ESTABLISHED | wc -l ## If this number is at or near your max_connections, the pool is exhausted ``` **Step 4 — Check for node-level issues** ```bash ## Is the node the pod is running on under pressure? kubectl describe node <node-name> | grep -A5 "Conditions:" ## Look for MemoryPressure, DiskPressure, or NetworkUnavailable ## Is the CNI (network plugin) dropping packets? kubectl get events -n production | grep -i "network\|timeout" ``` **Step 5 — Use distributed traces** If you have Jaeger or Zipkin set up, look at the traces for the failing 3% of requests. The trace will show you exactly where in the call chain the timeout is happening — whether it is the service itself, its database call, or a downstream service it is calling. **What to say in the interview:** "I would not immediately restart the pod. I would first capture the state — which pod, what its resource usage is, what its connection count is — because once you restart, the evidence is gone. Capture first, fix second." --- ### 5. Explain how HPA and VPA work and when you would use each. What happens if you use both at the same time? **What the interviewer is testing:** Scaling judgment. Many engineers know what HPA and VPA are but have not thought through the conflict. **Answer:** **HPA (HorizontalPodAutoscaler)** scales the number of pod replicas based on observed metrics. The default metric is CPU utilization, but you can scale on any custom metric via the metrics API. Traffic increases → CPU rises → HPA adds more replicas Traffic decreases → CPU drops → HPA removes replicas ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: payment-service-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: payment-service minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 ## scale up when average CPU hits 70% ``` **VPA (VerticalPodAutoscaler)** adjusts the CPU and memory requests/limits of individual pods over time, based on observed usage. If you set `requests: memory: 256Mi` but the container consistently uses 600Mi, VPA will update the pod spec to request 600Mi. VPA is most useful when you have a pod where traffic does not vary much but the right resource size is hard to know upfront — like a batch job, a sidecar, or a database. **The conflict when using both:** HPA scales replicas based on CPU utilization relative to the resource request. VPA changes the resource request. If VPA increases the memory request, Kubernetes needs to restart the pod (pod spec changes require a restart). That restart temporarily removes a replica, which might trigger HPA to add another. The two controllers fight each other. The official guidance (and what you should say in an interview) is: **do not use HPA and VPA on the same resource metric at the same time.** You can use HPA on CPU and VPA on memory — different metrics — and they coexist without conflict. Or use VPA in `Off` or `Initial` mode just to get recommendations without auto-applying them. > 📌 **Remember:** HPA is for stateless services where you handle load by running more copies. VPA is for pods where you need to right-size the resource allocation. Pick one based on the problem you are solving. --- ### 6. What is a PodDisruptionBudget and what breaks if you do not use one? **What the interviewer is testing:** Operational maturity. Many engineers deploy to Kubernetes without PDBs and have never had a node maintenance event to teach them why it matters. **Answer:** A **PodDisruptionBudget** tells Kubernetes the minimum number of pods of a given application that must be available at any time during a voluntary disruption — a node drain, a cluster upgrade, or a rolling deployment. Without a PDB, when you run `kubectl drain <node>` to take a node offline for maintenance, Kubernetes will evict every pod on that node as fast as it can. If all 3 replicas of your payment service happen to be on that node, all 3 are evicted simultaneously. Your service goes down during a maintenance window you planned. ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: payment-service-pdb spec: minAvailable: 2 ## at least 2 pods must be running at all times selector: matchLabels: app: payment-service ``` With this PDB in place, `kubectl drain` will only proceed when it can guarantee at least 2 payment-service pods remain running. If draining a node would bring the count below 2, the drain pauses and waits for a replacement pod to start on another node first. **The tradeoff:** A strict PDB can slow down or completely block node maintenance if the cluster does not have enough spare capacity to reschedule the pods. A PDB of `minAvailable: 3` on a 3-replica deployment running at full capacity will block drains indefinitely — there is nowhere to move the pods. The right approach is to set `minAvailable` to the minimum that keeps the service functional, not the maximum you want in normal operation. For a 3-replica service, `minAvailable: 2` is usually the right choice. > 🔴 **Common Mistake:** Setting `minAvailable` equal to the total replica count. This blocks all maintenance. Always leave at least one disruption slot. --- ### 7. Walk me through what happens inside Kubernetes when you scale a Deployment from 3 to 10 replicas. **What the interviewer is testing:** Whether you understand the control plane internals — the reconciliation loop. This separates engineers who have used Kubernetes from those who understand it. **Answer:** This is a multi-component orchestration that shows how the entire control plane works together. kubectl scale deployment payment-service --replicas=10 | v kubectl sends PATCH request to API Server API Server validates the request and your RBAC permissions API Server writes new desired state (replicas: 10) to etcd | v Deployment Controller (inside kube-controller-manager) is watching etcd for changes to Deployment objects It sees: desired=10, actual=3 → creates a ReplicaSet with 10 desired | v ReplicaSet Controller sees 7 pods need to be created It creates 7 new Pod objects in etcd (status: Pending) | v Scheduler watches for Pending pods with no node assignment For each pod it evaluates: resource requests, node capacity, taints/tolerations, affinity rules, topology spread constraints It writes a node assignment to each pod object in etcd | v kubelet on each assigned worker node watches etcd for pods assigned to its node It pulls the container image (if not cached) Starts the containers using the container runtime (containerd) Reports pod status back to the API Server | v Endpoints controller sees new Running pods with matching labels Updates the Service Endpoints object so traffic routes to new pods | v kube-proxy on each node updates iptables/ipvs rules New pods now receive traffic The key insight to mention in an interview: **nothing in Kubernetes calls anything directly.** Every component watches etcd for state changes and acts on them. This is why Kubernetes is resilient — if the scheduler crashes and restarts, it just reads current state from etcd and continues where it left off. --- ### 8. What is the difference between a Deployment rolling update and a blue/green deployment in Kubernetes? When would you choose each? **What the interviewer is testing:** Deployment strategy judgment. Junior engineers often treat "blue/green" as automatically better — the mature answer explains the real tradeoffs. **Answer:** **Rolling Update (Deployment default):** Kubernetes replaces old pods with new pods gradually — a few at a time, controlled by `maxSurge` and `maxUnavailable`. ```yaml strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 ## can temporarily have 2 extra pods above desired count maxUnavailable: 0 ## never have fewer pods than desired count ``` With this config on a 10-replica deployment, Kubernetes starts 2 new pods, waits for them to pass readiness checks, then terminates 2 old pods. Repeats until all 10 are updated. At no point are you below 10 available pods. Pros: No extra infrastructure needed. Traffic shifts gradually — a bad deploy affects a growing percentage of traffic, not 100% at once. Cons: Old and new versions run simultaneously during the rollout. If they are incompatible (different API contracts, different database schemas), this causes errors during the window. **Blue/Green:** Two complete environments. Blue is live. Green is the new version. You switch 100% of traffic at once. In Kubernetes, you implement this with separate Deployments and a Service that selects by label: ```yaml ## Traffic goes here kind: Service spec: selector: app: payment-service version: blue ## change this to 'green' to do the cutover --- ## Blue deployment — currently live kind: Deployment metadata: name: payment-service-blue spec: template: metadata: labels: app: payment-service version: blue --- ## Green deployment — new version, warmed up and ready kind: Deployment metadata: name: payment-service-green spec: template: metadata: labels: app: payment-service version: green ``` Pros: Clean cutover. You can test green fully before switching. Rollback is instant — just change the selector back. Cons: Requires double the compute during the transition. Both environments must be running simultaneously. **When to choose which:** * Use rolling update for most stateless services — it is simpler and uses less infrastructure. * Use blue/green when you have a schema change that makes old and new code incompatible during the transition window, when you need the ability to test the new version under real load before switching, or when instant rollback is a hard requirement. --- ### Observability and SRE ### 9. Your team wants to define SLOs for the first time. Walk me through how you would do it for an order service. **What the interviewer is testing:** Whether you have actually operationalized SLOs or just know the acronym. Companies with mature DevOps practices ask this at mid-level. Most candidates freeze here. **Answer:** An SLO (Service Level Objective) is a target for how well your service should perform, expressed as a percentage over a time window. It exists so your team has an agreed definition of "working" before an incident happens, not during one. **Step 1 — Identify what matters to the user** For an order service, the user cares about three things: * Can they place an order successfully? * Does the order page load fast enough? * Is the order data accurate? Translate these into measurable signals — the Google SRE book calls these the four golden signals: latency, traffic, errors, and saturation. **Step 2 — Write specific SLOs** ``` Availability SLO: 99.9% of order placement requests return a success response over a 28-day rolling window. (This allows ~43 minutes of downtime per month) Latency SLO: 95% of order placement requests complete in under 300ms. 99% complete in under 1000ms. Over a 28-day rolling window. ``` **Step 3 — Define the error budget** If your availability SLO is 99.9%, your error budget is 0.1% — that is the amount of failure you are allowed per month. Over 28 days: `28 * 24 * 60 * 0.001 = ~40 minutes` of allowed downtime. The error budget is the key to balancing reliability and velocity. If you have burned 80% of the month's error budget in week 2, that is the signal to slow down feature releases and focus on stability. If you are at 5% burned at month end, you have budget to take on riskier changes. **Step 4 — Instrument it in Prometheus** ```yaml ## Availability: track the ratio of successful requests ## A request is successful if it returns 2xx or 3xx - record: job:order_service_availability:ratio_rate5m expr: | sum(rate(http_requests_total{job="order-service",status=~"[23].."}[5m])) / sum(rate(http_requests_total{job="order-service"}[5m])) ## Alert when error budget burn rate is too fast - alert: OrderServiceErrorBudgetBurning expr: job:order_service_availability:ratio_rate5m < 0.999 for: 5m labels: severity: warning annotations: summary: "Order service availability SLO at risk" ``` **What to add unprompted in an interview:** "I would also define what counts as a valid SLI — which requests should be included. Health check requests from the load balancer should probably not count toward the SLO. A synthetic monitoring probe that hits the endpoint every 30 seconds should count. Getting this boundary right matters more than the math." --- ### 10. Explain what high cardinality means in Prometheus and why it is a problem. **What the interviewer is testing:** Real Prometheus operational experience. High cardinality is the most common reason Prometheus clusters run out of memory. Engineers who have not hit it in production often do not know about it. **Answer:** In Prometheus, every unique combination of label values creates a separate time series. Cardinality is the total number of unique time series being tracked. High cardinality means you have labels that can take a very large number of different values — which creates an explosion of time series. **The dangerous labels:** ```python ## This is fine — region has 5-10 possible values http_requests_total{service="orders", region="ap-south-1", status="200"} ## This is dangerous — user_id can be millions of values ## Each unique user_id creates a separate time series http_requests_total{service="orders", user_id="user-8472901", status="200"} ## This is catastrophic — request_id is unique per request http_requests_total{service="orders", request_id="550e8400-e29b-41d4-a716-446655440000"} ## One new time series per request = your TSDB fills up and OOMs in minutes ``` **Why it matters in production:** Prometheus stores all active time series in memory. Each time series uses roughly 3-4KB of memory. If you have 10 million time series (easy to hit with a user_id label on a service with 1 million users), that is 30-40GB of RAM just for the metric storage — before any actual data. When Prometheus runs out of memory, it crashes. Depending on your storage configuration, you may lose recent data. Your alerting goes dark. This is a production outage caused by a monitoring system that was supposed to prevent outages. **How to fix it:** * Never add high-cardinality values (user IDs, request IDs, session tokens, IP addresses) as label dimensions * If you need per-user data, put it in logs or traces, not metrics * Use `recording rules` to pre-aggregate high-cardinality data at collection time * Run `promtool tsdb analyze` periodically to find your highest-cardinality metrics ```bash ## Check your current series count curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.headStats.numSeries' ## Find the metrics with most series curl -s 'http://localhost:9090/api/v1/label/__name__/values' | jq '.data | length' ``` > 🔴 **Common Mistake:** Adding environment-specific details like pod names or container IDs as labels without thinking. `pod_name` sounds useful, but in a 100-pod deployment with rollouts, the pod names change constantly — each rollout creates new time series and orphans old ones. --- ### 11. You are designing an observability stack for a new microservices deployment with 30 services. Walk me through your choices and why. **What the interviewer is testing:** End-to-end observability thinking. This is the question the KORE1 guide specifically calls out as where strong candidates get eliminated — they name the tools without explaining the selection logic. **Answer:** Start with what you are trying to achieve, not what tools to install. You need to answer three questions about your system at any time: * Is it working? (metrics + alerting) * Why is it failing? (logs + traces) * Who is affected and how badly? (SLOs + error budgets) **Metrics — Prometheus + Grafana** Prometheus scrapes metrics from your services on a pull model. Every service exposes a `/metrics` endpoint. Prometheus reads it on a schedule. For 30 services, you will want: * One Prometheus instance per cluster (or use the operator pattern) * Recording rules to pre-aggregate expensive queries * AlertManager to route alerts to PagerDuty, Slack, or your on-call tool * Grafana for dashboards, pulling from Prometheus as the data source **Logs — Loki (if on Grafana stack) or ELK** If you are already using Grafana for metrics, Loki is the natural choice for logs — same query interface, same dashboard, and operationally much simpler than ELK. Loki only indexes labels (not the full log content), which makes it far cheaper to run. ELK (Elasticsearch, Logstash, Kibana) is more powerful for full-text search across structured logs but costs significantly more in compute and storage, and Elasticsearch requires careful tuning to stay healthy. ```bash ## Loki query — same syntax as PromQL (LogQL) {namespace="production", app="order-service"} |= "error" | json | status >= 500 ``` **Traces — OpenTelemetry + Tempo or Jaeger** For 30 services, distributed tracing is not optional — it is how you find which service in a chain is causing latency. Without traces, you are guessing. Use OpenTelemetry as the instrumentation standard. It is vendor-neutral — you instrument once and can send to Jaeger, Tempo, Zipkin, or a commercial tool without changing your code. For a self-hosted stack: Tempo (from Grafana) integrates natively with Grafana and Loki, so you can correlate a log line directly with its trace. **The integration that makes this valuable:** The three pillars become powerful when they connect. In Grafana, you can: 1. See an error rate spike in a metric dashboard 2. Click through to the logs for that time window 3. Find a specific log line with a trace ID 4. Click the trace ID to see the full distributed trace This workflow takes a potential 30-minute debugging session down to 3 minutes. **What to say about OpenTelemetry specifically:** "I would use OpenTelemetry's auto-instrumentation where possible — for Java, Python, and Go services, it instruments HTTP clients and servers automatically without changing application code. Manual spans are only needed for custom business operations that matter to trace." --- ### Infrastructure as Code ### 12. Your `terraform apply` fails halfway through. What do you do? **What the interviewer is testing:** Terraform state management under real conditions. This is the 2AM scenario they want to know you can handle. **Answer:** A partial apply is one of the most stressful IaC situations because your infrastructure is now in an inconsistent state — some resources were created, some were not, and the state file may or may not accurately reflect what actually exists. **Step 1 — Do not run terraform apply again immediately** Your first instinct might be to just re-run. Do not. Understand what state the apply left things in first. ```bash ## See what Terraform currently thinks exists terraform state list ## Compare state to reality for specific resources terraform plan ## This will show what Terraform thinks needs to happen to reach desired state ``` **Step 2 — Identify what was created before the failure** ```bash ## Check the apply output logs for the last successful resource creation ## Or check your cloud console — which resources exist? ## If a resource was created but Terraform did not record it in state: terraform import aws_instance.web i-1234567890abcdef0 ## This adds the existing resource to state without creating a new one ``` **Step 3 — Decide: fix forward or roll back** In most cases, **fix forward** is safer. You already have partial infrastructure. Trying to destroy it and start over can cause more damage than carefully applying the remaining changes. ```bash ## Target only the failed resource to continue terraform apply -target=aws_rds_instance.main ## This applies changes only to the specified resource and its dependencies ``` If the failure was in a destructive change (a database was being replaced and the old one was deleted but the new one failed to create), you have a data loss situation. This is why you should never do in-place replacements of stateful resources — use blue/green patterns at the infrastructure level too. **Step 4 — Communicate status while you work** This is what interviewers actually watch for. "I would immediately post in the on-call channel that we have a partial infrastructure state and services may be degraded. Do not deploy anything until this is resolved." **Prevention — what you say you would add:** ```hcl ## Add lifecycle rules to prevent accidental destruction of critical resources resource "aws_rds_instance" "main" { lifecycle { prevent_destroy = true ## Terraform will refuse to destroy this resource } } ``` And always use `-out` to save the plan and apply that exact plan: ```bash terraform plan -out=tfplan.out ## Review the plan carefully terraform apply tfplan.out ## Applies exactly what you reviewed ``` --- ### 13. Explain Terraform modules and when they make things worse instead of better. **What the interviewer is testing:** IaC maturity. Blind use of modules is a junior mistake — knowing when not to use them is a mid-level signal. **Answer:** A Terraform module is a reusable block of configuration — a folder of `.tf` files that can be called multiple times with different inputs. Modules enforce consistency and reduce duplication. ```hcl ## A module that creates a standard EKS cluster module "eks_cluster" { source = "./modules/eks" cluster_name = "production" cluster_version = "1.29" node_count = 3 instance_type = "t3.xlarge" region = "ap-south-1" } ## Same module, different inputs for staging module "eks_cluster_staging" { source = "./modules/eks" cluster_name = "staging" cluster_version = "1.29" node_count = 1 instance_type = "t3.medium" region = "ap-south-1" } ``` **When modules are the right choice:** * Infrastructure patterns that repeat across environments (VPCs, EKS clusters, RDS instances with standard configurations) * Enforcing security baselines — all VPCs must have flow logs, all S3 buckets must be private — by building that into the module * Large teams where different people manage different parts of infrastructure **When modules make things worse:** *Over-abstraction.* A module that wraps a single `aws_s3_bucket` resource adds complexity without value. The module input variables become a reinvention of the resource's own arguments, worse. *Version drift.* If you are using a module from a public registry (Terraform Registry or GitHub) and you do not pin the version, a module update can change your infrastructure on the next plan. Pin your module sources: ```hcl module "vpc" { source = "terraform-aws-modules/vpc/aws" version = "5.1.2" ## always pin — never leave this floating } ``` *Debugging complexity.* When a module fails, the error points into the module's internals. If the module has three levels of nesting, finding the actual resource that caused the error requires unwrapping multiple layers. *Inflexibility.* A module built for 80% of your use cases will eventually need to handle the 20%. Adding every edge case as a module variable makes the module interface as complex as the underlying resources — except now you have an indirection layer too. The right signal for "should this be a module" is: will I use this exact pattern at least 3 times, and does it have meaningful internal logic that should be hidden from callers? --- ### Networking and Security ### 14. What is mTLS and how does it work in a microservices environment? What problem does it solve that TLS alone does not? **What the interviewer is testing:** Security depth beyond the basics. mTLS is directly relevant to service mesh adoption and zero-trust architecture, which comes up at any company with mature microservices. **Answer:** Standard TLS (one-way TLS) is what your browser uses when it visits any HTTPS website. The server presents a certificate. The client verifies it. The client is anonymous. Client → "I want to connect to api.company.com" Server → presents certificate (proves it is api.company.com) Client → verifies certificate, trusts the server Encrypted connection established **The problem:** In a microservices environment, the server knows it is talking to someone with a valid encrypted connection, but it does not know which service is calling it. Any service — or any attacker who has gained access to the internal network — can call the payment service. **mTLS (mutual TLS)** adds certificate authentication in both directions. The client also presents a certificate, and the server verifies it. Service A → presents its own certificate (proves it is Service A) Service B → presents its certificate (proves it is Service B) Both verify each other's certificates Encrypted connection established with mutual identity proof Now the payment service can enforce: "I will only accept connections from the order service and the auth service. If anything else tries to connect, I reject it at the TLS level before any application code runs." **How a service mesh implements this:** Manually managing certificates for 30 services is operationally nightmarish — rotating certificates, distributing them to the right pods, handling renewals. This is exactly what service meshes like Istio and Linkerd automate. The mesh injects a sidecar proxy (Envoy in Istio's case) into every pod. The control plane issues short-lived certificates to each sidecar automatically. mTLS between services happens at the sidecar level — your application code does not know it is happening. ```bash ## In Istio, enable strict mTLS for an entire namespace kubectl apply -f - <<EOF apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: production spec: mtls: mode: STRICT ## reject any non-mTLS traffic EOF ``` **What mTLS does not solve:** It proves that Service A has the right certificate, not that Service A is authorized to perform a specific action. For that you still need application-level authorization (checking if the calling service has permission to access a particular endpoint). mTLS is identity, not permissions. --- ### 15. A developer accidentally committed AWS credentials to a public GitHub repository 10 minutes ago. You get notified. Walk me through your incident response in order. **What the interviewer is testing:** Security incident response speed and sequence. The correct order matters — most candidates know to revoke the key but do not move fast enough or in the right order. **Answer:** This is a race. GitHub's ecosystem and malicious bots actively scan public repositories for credentials in near real-time. By the time you are notified at 10 minutes, the key may already have been found. **Minute 0-2: Disable the key immediately** ```bash ## Get the access key ID from the commit (check the git history) git log --patch -- path/to/committed/file | grep -i "AKIA" ## Disable it — this is faster than deleting aws iam update-access-key \ --access-key-id AKIAIOSFODNN7EXAMPLE \ --status Inactive \ --user-name developer-name ``` Do this before anything else. An inactive key cannot be used even if it was already found. **Minute 2-5: Investigate what happened with the key** ```bash ## Check CloudTrail for any API calls made with this key aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \ --start-time $(date -d "2 hours ago" --iso-8601=seconds) ## Look specifically for: ## - New IAM users or access keys created (attacker creating persistence) ## - EC2 instances launched (cryptomining) ## - S3 data accessed or exfiltrated ## - Lambda functions created (attacker infrastructure) ## - CloudFormation stacks (attacker deploying their own infrastructure) ``` **Minute 5-10: Generate a new key, update all systems** ```bash ## Create a replacement key aws iam create-access-key --user-name developer-name ## Update every system using the old key: ## - CI/CD environment variables ## - Application config (Secrets Manager, Vault) ## - Local developer machines ## Then permanently delete the old key (not just disable) aws iam delete-access-key \ --access-key-id AKIAIOSFODNN7EXAMPLE \ --user-name developer-name ``` **Remove from GitHub — but understand this does not help security** Even after you rewrite Git history to remove the key, GitHub's API indexers have already scanned it. Any bot that found it has the credentials stored. Removal is good hygiene but treat the key as fully compromised regardless of whether you cleaned up the history. **After the incident — the prevention layer:** ```bash ## Add to every developer's pre-commit hook pip install detect-secrets detect-secrets scan > .secrets.baseline ## This blocks commits containing high-entropy strings that look like credentials ## Enable GitHub Advanced Security (secret scanning) on the org ## This scans every push for credential patterns and alerts immediately ``` > 📌 **Remember:** The correct answer to "how should EC2 instances access AWS services" is IAM Instance Roles, not access keys. This whole incident class is prevented by never using long-lived access keys on infrastructure. Mention this at the end of your answer. --- ### Docker and Containers — Advanced ### 16. What is a multi-stage Docker build and why does it matter in production? **What the interviewer is testing:** Image optimization and production best practices. Every serious DevOps role expects this. **Answer:** A multi-stage build uses multiple `FROM` statements in a single Dockerfile. Each `FROM` starts a new stage. You can copy files from one stage to the next — but only the files you explicitly copy, not everything in the build environment. The problem it solves: build tools (compilers, test frameworks, package managers) are needed to build your application but should never be in your production image. They add size, attack surface, and potential vulnerabilities. **Without multi-stage (naive approach):** ```dockerfile FROM golang:1.21 ## golang:1.21 is 800MB WORKDIR /app COPY . . RUN go build -o server . ## Final image: 800MB base + your app + all Go toolchain CMD ["./server"] ``` Your production image ships the entire Go compiler, standard library sources, and build tools — none of which your running application needs. **With multi-stage:** ```dockerfile ## Stage 1 — build environment FROM golang:1.21 AS builder WORKDIR /app COPY go.mod go.sum ./ RUN go mod download ## cache dependencies separately COPY . . RUN CGO_ENABLED=0 go build -o server . ## static binary, no external dependencies ## Stage 2 — runtime environment ## scratch is literally empty — no OS, no shell, no package manager FROM scratch COPY --from=builder /app/server /server COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ CMD ["/server"] ``` The final image: your binary (~10-20MB) plus CA certificates. Nothing else. No shell, no package manager, no way for an attacker to run arbitrary commands even if they find a vulnerability in your app. For Python or Node.js (interpreted languages where you cannot produce a static binary): ```dockerfile ## Stage 1 — install all deps including dev deps for building FROM python:3.11 AS builder WORKDIR /app COPY requirements.txt . RUN pip install --user -r requirements.txt ## Stage 2 — copy only the installed packages, not the build tools FROM python:3.11-slim COPY --from=builder /root/.local /root/.local COPY src/ ./src/ ENV PATH=/root/.local/bin:$PATH CMD ["python", "src/main.py"] ``` `python:3.11-slim` is 45MB versus `python:3.11` at 1.1GB. The `--from=builder` copy brings over only the installed packages. **What to mention about security:** "Smaller images have a smaller attack surface. Every package that is not in the image is a CVE that cannot affect your container. I would run Trivy against the final image in the CI pipeline and block on CRITICAL findings." --- ### 17. What is the difference between `docker stop`, `docker rm`, and what happens to volumes when you remove a container? **What the interviewer is testing:** Container lifecycle understanding. Volumes are where many engineers lose data by accident. **Answer:** **`docker stop`** sends SIGTERM to the main process, waits a grace period (default 10s), then sends SIGKILL if still running. The container is stopped but its filesystem still exists. You can `docker start` it again and it resumes from where it was. **`docker rm`** deletes the container's writable layer — the filesystem that sits on top of the image layers. This removes all files written inside the container that were not stored in a volume. The container is gone permanently. ```bash ## Stop then remove in sequence docker stop my-container docker rm my-container ## Or stop and remove in one command docker rm -f my-container ## -f force-removes even if running (sends SIGKILL) ``` **What happens to volumes:** By default, `docker rm` does NOT remove volumes attached to the container. Named volumes persist until explicitly deleted. ```bash ## This removes the container but the 'postgres-data' volume still exists docker rm postgres-container ## To remove both container and its volumes: docker rm -v postgres-container ## removes anonymous volumes only ## Named volumes still survive even with -v ## Explicitly remove a named volume docker volume rm postgres-data ## Remove all unused volumes (careful in production) docker volume prune ``` The distinction between anonymous and named volumes: ```dockerfile ## Anonymous volume — created at container start, random ID VOLUME /var/lib/postgresql/data ## Named volume — you explicitly create it, it persists by name docker run -v postgres-data:/var/lib/postgresql/data postgres ``` Named volumes are the correct approach for any data you care about. Anonymous volumes are fine for temporary data that does not need to outlive the container. > 🔴 **Common Mistake:** Running `docker system prune` without realizing it removes stopped containers and dangling volumes. In production, this can delete database data. Always check `docker volume ls` before running any prune command. --- ### AWS and Cloud Architecture ### 18. Design the networking architecture for a production application on AWS. It needs a web tier, application tier, and database tier with appropriate security boundaries. **What the interviewer is testing:** AWS networking depth — VPC design, subnet architecture, security groups. This is a whiteboard question that separates junior from mid-level thinking. **Answer:** The principle is defense in depth — each tier can only talk to the tier it needs, through the ports it needs, and nothing else. Internet | v Internet Gateway | v Public Subnet (AZ-a and AZ-b) - Application Load Balancer (internet-facing) - NAT Gateway (for private subnets to reach internet) | v Private Subnet — App Tier (AZ-a and AZ-b) - EC2 instances / EKS worker nodes - Auto Scaling Group | v Private Subnet — Data Tier (AZ-a and AZ-b) - RDS Multi-AZ (primary in AZ-a, standby in AZ-b) - ElastiCache (Redis) **Security groups — the key detail:** Security groups are the firewall rules that enforce tier separation. ``` ALB Security Group: Inbound: 443 from 0.0.0.0/0 (internet HTTPS) 80 from 0.0.0.0/0 (redirect to HTTPS) Outbound: 8080 to App SG (forward to app tier) App Tier Security Group: Inbound: 8080 from ALB SG (only from the load balancer, not internet) Outbound: 5432 to DB SG (PostgreSQL) 6379 to Cache SG (Redis) 443 to 0.0.0.0/0 (for calling external APIs, via NAT Gateway) Database Security Group: Inbound: 5432 from App SG (only from app tier) Outbound: None needed (databases do not initiate connections) ``` The security groups reference each other by group ID, not by IP. This means even if you add more app tier instances or scale down, you never have to update the database security group — it always allows the App Tier SG. **Why two AZs for every tier:** Every subnet is duplicated across two availability zones. If AWS AZ-a has an outage (happens rarely but does happen), your ALB automatically routes to AZ-b, the app tier has instances in AZ-b, and the RDS standby in AZ-b automatically promotes to primary. Your service continues with a brief interruption. **What to mention about the NAT Gateway:** NAT Gateway allows instances in private subnets to make outbound internet connections (to download packages, call external APIs) without being reachable from the internet. It sits in the public subnet and routes outbound traffic for private instances. NAT Gateway is priced by data transfer — at high volume it becomes expensive. For EC2 instances downloading S3 data, use a VPC Endpoint for S3 instead (free and keeps traffic inside AWS). --- ### 19. What is the difference between AWS IAM roles and IAM users? Why should your applications never use IAM users? **What the interviewer is testing:** IAM security best practices. Misuse of IAM users is one of the most common AWS security vulnerabilities. **Answer:** **IAM Users** have long-lived credentials — an access key ID and secret access key that do not expire until you explicitly rotate or delete them. They are designed for human console access or legacy use cases. **IAM Roles** are assumable identities with temporary credentials. When a service assumes a role, it gets a temporary access key, secret key, and session token that expire after 1-12 hours (configurable). The STS (Security Token Service) issues these automatically. **Why applications should never use IAM users:** Long-lived credentials are dangerous for three reasons: 1. **They can be stolen and used indefinitely.** An access key committed to Git, leaked in logs, or copied to a developer's laptop remains valid until someone manually rotates it. This is the source of the vast majority of AWS account compromises. 2. **Rotation is operationally painful.** Rotating an access key requires updating it everywhere it is used — CI environment variables, application config, servers. Teams avoid doing it. Keys end up being years old. 3. **No automatic expiry.** Even if a key is stolen, the attacker has unlimited time to use it. Temporary credentials from STS expire — even if stolen, they become useless within hours. **The alternatives:** *For EC2 instances:* IAM Instance Role — credentials are available via the instance metadata service and rotate automatically. *For EKS pods:* IRSA (IAM Roles for Service Accounts) — assigns an IAM role to a Kubernetes service account. Pods using that service account get temporary credentials via the AWS SDK automatically. ```yaml ## Annotate the Kubernetes service account with the IAM role ARN apiVersion: v1 kind: ServiceAccount metadata: name: payment-service annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payment-service-role ``` *For CI/CD pipelines:* OIDC federation — GitHub Actions or GitLab CI assume an IAM role directly using an OIDC token. No static credentials stored anywhere. *For Lambda:* Execution role — Lambda assumes the role automatically when the function runs. **The only legitimate use for IAM users:** Human console access combined with MFA, or legacy tooling that genuinely cannot use roles. In all other cases, roles are the correct answer. --- ### Incident Management and On-Call ### 20. You are on-call. At 11 PM you get paged — the checkout service error rate has jumped from 0.1% to 8%. Walk me through your complete response. **What the interviewer is testing:** On-call maturity and incident response process. Companies want to see that you are systematic, that you communicate, and that you do not make things worse by making changes under pressure. **Answer:** **0-2 minutes: Acknowledge and communicate** Acknowledge the alert immediately. Post in the incident channel: > "Acknowledged checkout service error rate spike — 0.1% to 8%. Starting investigation. Do not push any deployments until further notice." Code freeze is the first action. The worst time for a deploy is during an active incident. **2-5 minutes: Establish a timeline** ```bash ## When did this start? Check when the metric crossed the threshold ## Look at Grafana — was there a deployment, a traffic spike, or an external event? ## Recent deployments kubectl rollout history deployment/checkout-service -n production ## Recent events kubectl get events -n production --sort-by='.lastTimestamp' | tail -30 ``` If a deployment happened in the last 30 minutes before the incident, that is your primary suspect. **5-10 minutes: Identify the error class** ```bash ## What kind of errors? 5xx? Timeouts? ## In Prometheus: rate(http_requests_total{service="checkout",status=~"5.."}[5m]) by (status) ## Is it 500 (application error) or 504 (timeout to downstream)? ## Check the logs kubectl logs -l app=checkout-service -n production --tail=100 | grep "ERROR\|FATAL\|panic" ## Previous logs if pods have restarted kubectl logs -l app=checkout-service -n production --previous --tail=50 ``` **10-15 minutes: Determine if you need to rollback** If the incident started immediately after a deployment and errors are clearly application-level: ```bash ## Rollback to previous known-good version kubectl rollout undo deployment/checkout-service -n production ## Watch the rollback kubectl rollout status deployment/checkout-service -n production ``` If no recent deployment, check the downstream services — is the payment gateway timing out? Is the database connection pool exhausted? ```bash ## Is the database reachable from the app? kubectl exec -it $(kubectl get pods -l app=checkout-service -n production -o jsonpath='{.items[0].metadata.name}') \ -- nc -zv db.internal 5432 ## Connection pool status (application-specific) kubectl logs -l app=checkout-service | grep "pool\|connection" | tail -20 ``` **Throughout: keep stakeholders updated every 10 minutes** > "Update at 23:15 — traced to database connection pool exhaustion. Rolling back last deployment which changed connection pool config. ETA 5 minutes." **After resolution: write the incident timeline while it is fresh** * When alert fired * When you acknowledged * Timeline of what you checked and found * What the root cause was * What you changed to fix it This becomes the post-mortem document reviewed on Monday. **What makes this answer stand out in an interview:** Mentioning the code freeze first, the communication cadence throughout, and writing the timeline immediately after resolution — not just the technical debugging steps. --- ### Advanced Topics ### 21. What is a service mesh and when does adding one make sense? What are the costs? **What the interviewer is testing:** Architecture judgment on complex infrastructure decisions. Adding a service mesh too early is a common over-engineering mistake. Not knowing when you need one is also a problem. **Answer:** A service mesh is an infrastructure layer that handles communication between services — traffic management, mTLS, observability, retries, circuit breaking — without requiring application code changes. Istio and Linkerd are the two most common options. It works by injecting a sidecar proxy (Envoy in Istio's case) into every pod. All traffic in and out of each pod flows through this proxy. The control plane configures all proxies centrally. Service A → Envoy sidecar → network → Envoy sidecar → Service B ↕ ↕ Istio control plane (issues certs, sets routing rules) **What it gives you:** * mTLS between all services with automatic certificate rotation — zero application code changes required * Distributed tracing via automatic span creation at every service boundary * Traffic splitting — send 10% of traffic to a new version while 90% goes to stable (canary at the mesh level, not the Kubernetes replica level) * Circuit breaking — stop sending traffic to a service that is failing repeatedly * Retries and timeouts — configurable per route without changing application code * Detailed per-service metrics automatically **When it makes sense:** * You have 20+ services and the operational complexity of managing security policies, retries, and observability per-service is becoming unsustainable * You have a compliance requirement that all inter-service traffic must be encrypted and authenticated * You want traffic management capabilities (canary, A/B testing) at a granularity that Kubernetes Services cannot provide **The real costs — and why you should not add one prematurely:** *CPU and memory overhead.* Every pod now runs two containers — your app and the Envoy sidecar. Envoy uses 50-200MB of memory per pod and adds measurable CPU overhead to every request. For a 100-pod cluster, you are paying for 100 extra Envoy instances. *Debugging complexity.* When something breaks, you now have to determine: is this a failure in my application, or in the Envoy proxy, or in the Istio control plane? The blast radius of a misconfigured mesh policy is every service in the mesh simultaneously. *Operational overhead.* Istio upgrades are non-trivial. The CRD surface area is large. Debugging network policies requires understanding of DestinationRules, VirtualServices, and PeerAuthentication objects. **The honest answer for an interview:** "For a team running 10 services, I would add mTLS with cert-manager and external-dns before reaching for Istio. The value of a full service mesh comes at 30+ services where the per-service configuration maintenance cost is genuinely higher than the mesh overhead." --- ### 22. You are asked to improve your organization's deployment process to reduce the mean time to recover (MTTR) from failed deployments. What changes do you make? **What the interviewer is testing:** Reliability engineering thinking — not just deploying faster but recovering faster. **Answer:** MTTR is determined by three things: time to detect a problem, time to decide to roll back, and time to execute the rollback. Every improvement targets one of these. **Reduce time to detect — better alerting and deployment monitoring** Most teams find out about a bad deployment from users, not monitoring. Fix this first. Add a deployment verification step that runs immediately after every production deploy: ```bash ## Smoke tests that run automatically after deployment ## These should complete in under 2 minutes and test the critical user paths ## Is the service returning 200? curl -sf https://api.company.com/health | jq '.status == "ok"' ## Is the critical endpoint working? curl -sf -X POST https://api.company.com/checkout \ -H "Content-Type: application/json" \ -d '{"test": true}' | jq '.success == true' ## Did error rate spike vs baseline? ## Check Prometheus alert within 2 minutes of deployment ``` Add a deployment marker to Grafana. When you deploy, draw a vertical line on every dashboard at that timestamp. The moment error rate changes, the correlation to the deployment is visually immediate. **Reduce time to decide — auto-rollback with clear criteria** Manual rollback decisions waste time. Define the criteria upfront: ```yaml ## ArgoCD progressive delivery with automatic rollback ## Using Argo Rollouts spec: strategy: canary: analysis: templates: - templateName: error-rate-check args: - name: service-name value: checkout-service ## Automatically rollback if analysis fails autoPromotionEnabled: false ``` The rule: if error rate or latency SLO is breached within 5 minutes of a deployment, automatically roll back without waiting for human decision. **Reduce time to execute — one-command rollback** ```bash ## Kubernetes rolling update — rollback is instant kubectl rollout undo deployment/checkout-service -n production ## This takes 30-60 seconds ## Make sure this is documented, tested regularly, and every on-call engineer knows it ## Not knowing the rollback command at 2 AM costs you 5-10 minutes ``` **The process change that matters most:** Run a quarterly "game day" where you deliberately deploy a bad version and practice detecting and rolling back. Most teams discover in these exercises that the rollback command is not documented, the smoke tests do not run automatically, or the deployment marker was never added to the dashboard. Discover this in a game day, not in a production incident. --- ### 23. What is the difference between a liveness probe and a readiness probe in Kubernetes? What happens if you configure them wrong? **What the interviewer is testing:** Most engineers know what these probes are. Fewer understand the failure consequences — which is where production incidents come from. **Answer:** Both probes run on a schedule and check whether a container is healthy. The difference is what Kubernetes does when they fail. **Readiness probe** — if this fails, Kubernetes removes the pod from the Service endpoints. Traffic stops routing to it. The pod keeps running but receives no requests. When the probe passes again, it gets added back. Use this for: application startup (pod is running but not ready yet), temporary overload, warming up a cache. **Liveness probe** — if this fails, Kubernetes kills and restarts the container. Use this for: detecting deadlocks or stuck states where the application process is alive but not making progress. ```yaml livenessProbe: httpGet: path: /health/live ## must return 200 if the process is functional port: 8080 initialDelaySeconds: 30 ## wait 30s after container starts before first check periodSeconds: 10 ## check every 10s failureThreshold: 3 ## restart after 3 consecutive failures readinessProbe: httpGet: path: /health/ready ## return 200 only when ready to serve traffic port: 8080 initialDelaySeconds: 5 periodSeconds: 5 failureThreshold: 3 ``` **What breaks when configured wrong:** *Liveness probe too aggressive (low initialDelaySeconds):* The app takes 45 seconds to start. Liveness probe starts at 10 seconds, sees a failure, restarts the container. The container never finishes starting. This is a CrashLoopBackOff that is entirely caused by the probe — the app code is fine. *Liveness probe pointing at a dependency:* Your liveness probe calls `/health` which checks database connectivity. Your database goes down briefly. Every pod in the deployment gets restarted simultaneously because they all fail the liveness check. You now have zero running pods during a database blip that would have been survivable if the pods had stayed up. *No readiness probe at all:* Kubernetes routes traffic to new pods immediately after the container starts — before the application has finished loading. The first real requests hit an app that is not ready. Users see errors during every deployment. > 📌 **Remember:** Liveness and readiness should check different things. Liveness checks "is this process functional." Readiness checks "is this process ready to serve traffic." Your liveness probe should never check external dependencies — only the health of the local process. --- ### 24. How does Kubernetes handle secret storage and what are its security limitations? What is the better alternative? **What the interviewer is testing:** Security maturity around secrets management. This is a mid-level filter question at security-conscious companies. **Answer:** Kubernetes Secrets store data as base64-encoded strings in etcd. They are not encrypted by default — base64 is encoding, not encryption. Anyone with access to etcd (or a backup of etcd) can read every Secret in plain text. ```bash ## A Secret that looks protected: kubectl get secret db-credentials -o yaml ## apiVersion: v1 ## kind: Secret ## data: ## password: cGFzc3dvcmQxMjM= ← this is just base64 ## Decode it trivially: echo "cGFzc3dvcmQxMjM=" | base64 --decode ## password123 ``` **The limitations of Kubernetes Secrets:** * No rotation — changing a secret requires manually updating the Secret object and restarting pods * No audit trail — you cannot easily see who read a secret or when * No expiry — secrets do not expire automatically * etcd encryption at rest is opt-in and not enabled by default in most clusters * Any pod with the right RBAC permissions can read any Secret in its namespace **The better alternative — external secrets managers:** Use AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager as the source of truth. Sync them into Kubernetes using External Secrets Operator (ESO): ```yaml ## ExternalSecret object — fetches from AWS Secrets Manager apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-credentials spec: refreshInterval: 1h ## re-sync every hour (picks up rotations) secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore target: name: db-credentials ## creates a Kubernetes Secret with this name data: - secretKey: password remoteRef: key: prod/db/credentials property: password ``` The actual secret value lives in AWS Secrets Manager where you have: encryption at rest (KMS), audit logs in CloudTrail, automatic rotation, and fine-grained IAM access control. The Kubernetes Secret is just a synchronized copy. **What to say at the end:** "I would also enable envelope encryption for etcd in the cluster config, even if using external secrets, because it is a cheap additional layer for any secrets that do land in etcd." --- ### 25. What is a Helm chart and when does Helm make your life harder instead of easier? **What the interviewer is testing:** Tool judgment. Over-reliance on Helm is a common mid-level anti-pattern. **Answer:** Helm is a package manager for Kubernetes. A Helm chart is a collection of Kubernetes manifests with Go templating — values can be substituted at deploy time, making the same chart work for dev, staging, and production with different configs. ```bash ## Deploy with environment-specific values helm upgrade --install payment-service ./charts/payment \ --namespace production \ --values values-production.yaml \ --set image.tag=v2.3.1 ## Rollback to previous release helm rollback payment-service 1 ``` **When Helm is the right choice:** * Deploying third-party software (databases, monitoring tools, ingress controllers) where the chart already exists and is maintained * Applications with genuinely different configurations across 3+ environments where managing separate manifests causes real duplication * Teams where templating and packaging standards improve consistency across multiple services **When Helm makes things harder:** *Debugging is harder.* When `helm upgrade` fails, the error often points into the rendered template, not your values file. You have to run `helm template` to see what was actually generated, then find where in your values the problem originated. ```bash ## Render the chart locally to debug what Helm actually generates helm template payment-service ./charts/payment \ --values values-production.yaml \ > rendered.yaml ## Now you can read and validate the actual YAML that would be applied ``` *Chart drift.* If you install a Helm chart and then someone does `kubectl edit` on a resource Helm manages, the next `helm upgrade` will overwrite that manual change silently. Helm owns what it deploys — manual edits outside Helm break this contract. *Simple services do not need it.* A single-service deployment with a Deployment, Service, and ConfigMap — three files totalling 80 lines — does not benefit from Helm templating. Adding Helm adds a values.yaml, a Chart.yaml, a templates directory, and dependency on the Helm binary. The honest mid-level answer: use Helm for shared infrastructure and third-party charts. For your own services, evaluate whether the templating complexity is actually paying for itself. --- ### 26. What is container image scanning and where in your pipeline does it belong? **What the interviewer is testing:** DevSecOps awareness. Security shifted left into the pipeline is expected at mid-level. **Answer:** Container image scanning checks your Docker image against a database of known vulnerabilities (CVEs) in the OS packages, language runtimes, and libraries your image contains. Tools: Trivy (most common), Snyk, Grype, AWS ECR native scanning. **Where it belongs — and why order matters:** ``` Build image → Scan image → Push to registry → Deploy ↑ Block here if CRITICAL CVEs found ``` Scan before push. If you scan after push or after deploy, vulnerable images are already in your registry or running in production. The entire point is to catch issues before they are deployed. ```yaml ## GitHub Actions example - name: Build image run: docker build -t myapp:${{ github.sha }} . - name: Scan image for vulnerabilities run: | trivy image \ --exit-code 1 \ ## fail the pipeline on findings --severity CRITICAL,HIGH \ ## block on CRITICAL and HIGH --ignore-unfixed \ ## skip CVEs with no fix available (noise reduction) myapp:${{ github.sha }} - name: Push to registry if: success() ## only push if scan passed run: docker push myapp:${{ github.sha }} ``` **What to configure to avoid false positives:** * `--ignore-unfixed` skips CVEs that have no patch available yet — you cannot fix them, they just create noise * Set severity threshold to CRITICAL only for strict pipelines, CRITICAL,HIGH for balanced * Create a `.trivyignore` file for known false positives with justification comments **Beyond the pipeline — continuous scanning:** The pipeline scan catches vulnerabilities at build time. New CVEs are published every day against images you built months ago. Add continuous registry scanning: ECR has this built in, or use Trivy Operator in Kubernetes which scans running pods and reports findings as Kubernetes resources. ```bash ## Check Trivy Operator findings in your cluster kubectl get vulnerabilityreports --all-namespaces ``` --- ### 27. What is a circuit breaker pattern and how would you implement it for a service that calls an unreliable external API? **What the interviewer is testing:** Resilience patterns. This is an architecture question that shows whether you think about failure modes, not just happy paths. **Answer:** A circuit breaker prevents your service from continuously hammering a failing downstream dependency. Without it, if the payment gateway is down, every request to your checkout service waits the full timeout period (say 30 seconds) before failing. Under load, you exhaust your thread pool waiting on a dependency that is not responding. Your checkout service goes down because the payment gateway is down — even if checkout could have degraded gracefully. The circuit breaker has three states: CLOSED (normal) → requests pass through to the external API | | failure rate exceeds threshold (e.g. 50% failures in 10s window) v OPEN (tripped) → requests immediately return an error without calling the API | | after a timeout (e.g. 30 seconds), allow one test request v HALF-OPEN (testing) → if test request succeeds → back to CLOSED → if test request fails → back to OPEN **Implementation with resilience4j (Java) or the pattern in Python:** ```python import time from enum import Enum class CircuitState(Enum): CLOSED = "closed" OPEN = "open" HALF_OPEN = "half_open" class CircuitBreaker: """ Wraps calls to an external service. Opens after failure_threshold consecutive failures. Retests after recovery_timeout seconds. """ def __init__(self, failure_threshold=5, recovery_timeout=30): self.state = CircuitState.CLOSED self.failure_count = 0 self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.last_failure_time = None def call(self, func, *args, **kwargs): if self.state == CircuitState.OPEN: ## Check if recovery timeout has elapsed if time.time() - self.last_failure_time > self.recovery_timeout: self.state = CircuitState.HALF_OPEN else: raise Exception("Circuit breaker OPEN — fast fail, not calling external API") try: result = func(*args, **kwargs) self._on_success() return result except Exception as e: self._on_failure() raise e def _on_success(self): self.failure_count = 0 self.state = CircuitState.CLOSED def _on_failure(self): self.failure_count += 1 self.last_failure_time = time.time() if self.failure_count >= self.failure_threshold: self.state = CircuitState.OPEN ``` **In production with a service mesh:** Istio implements circuit breaking at the proxy level with no application code changes: ```yaml apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: payment-gateway spec: host: payment-gateway.external.com trafficPolicy: outlierDetection: consecutive5xxErrors: 5 ## trip after 5 consecutive 5xx responses interval: 10s ## evaluated over a 10s window baseEjectionTime: 30s ## eject the host for 30s ``` **What to say in the interview:** "The circuit breaker is paired with a fallback — what do you do when the circuit is open? For a payment service the answer might be 'queue the request for retry.' For a recommendation service it might be 'return cached results.' Having no fallback means the circuit breaker just moves the error, not resolves it." --- ### 28. How do you handle rollbacks in a GitOps workflow where the deployment is triggered by a Git commit? **What the interviewer is testing:** GitOps operational depth. Many engineers understand the happy path but not the rollback story. **Answer:** In a GitOps workflow, Git is the source of truth. The correct rollback is a Git operation — not a direct `kubectl rollout undo`. Running `kubectl rollout undo` outside of Git means your live cluster state diverges from what Git says it should be. ArgoCD will detect the drift and reconcile it back — undoing your manual rollback. **The correct rollback flow:** ```bash ## Option 1 — Revert the commit that triggered the bad deployment git log --oneline ## a3f8d21 update image tag to v2.4.0 ← bad deployment ## b1c9e44 update image tag to v2.3.9 ← last known good git revert a3f8d21 ## creates a new commit that undoes the image tag change git push origin main ## ArgoCD detects the change and deploys v2.3.9 ``` ```bash ## Option 2 — If you use a separate config/gitops repo (common pattern) ## CI writes image tag to the gitops repo ## Rollback means reverting that commit in the gitops repo cd gitops-repo git log --oneline deployments/payment-service/ ## Revert the last tag update git revert HEAD git push ## ArgoCD syncs within 3 minutes (or immediately if you trigger it) ``` **Speed matters — ArgoCD manual sync:** Auto-sync has a polling interval (usually 3 minutes). If the incident is severe and you cannot wait 3 minutes: ```bash ## Force an immediate sync after your git revert argocd app sync payment-service ## Deployment starts within seconds ``` **The pattern that makes rollbacks fast — separate image tag from everything else:** Keep your application code repo and your Kubernetes manifest repo separate. The CI pipeline only changes one line — the image tag — in the manifests repo. A rollback is reverting one line in one file. This is fast, auditable, and safe. ``` application-repo/ ← developer code, CI builds image here gitops-repo/ deployments/ payment-service/ deployment.yaml ← CI auto-updates image.tag here only ``` **What to say in the interview:** "The key principle is: you never bypass Git in a GitOps setup. Even in an incident. If you run `kubectl rollout undo` manually, your next ArgoCD sync will re-apply the broken version from Git and undo your fix. The fix always goes through Git first." --- ### 29. What is the difference between horizontal and vertical scaling for a Kubernetes cluster itself — adding nodes versus resizing nodes? When does each make sense? **What the interviewer is testing:** Cloud infrastructure cost and scaling judgment at the cluster level, not just pod level. **Answer:** **Horizontal cluster scaling (add more nodes):** Add worker nodes to the cluster. Each new node brings its own CPU, memory, and pod capacity. This is what Cluster Autoscaler does — watches for pending pods that cannot schedule due to insufficient resources, and adds nodes to accommodate them. ```bash ## Cluster Autoscaler adds nodes automatically when pods are Pending kubectl get pods -n kube-system | grep cluster-autoscaler ## Check current node count kubectl get nodes ``` **Vertical cluster scaling (resize existing nodes):** Replace existing nodes with larger instance types — upgrading from `t3.xlarge` (4 vCPU, 16GB) to `c5.2xlarge` (8 vCPU, 16GB). In AWS this means creating a new node group with the larger instance type and draining the old nodes into it. **When to choose horizontal:** * Your workload is stateless and scales by adding replicas * You want fault tolerance — more nodes means one node failure has smaller impact * Your pods have moderate resource needs and many of them fit on standard nodes * You need to scale up and down quickly — Cluster Autoscaler can add nodes in 2-3 minutes **When to choose vertical (larger nodes):** * You have pods with large memory requirements that do not fit on standard nodes — ML model serving, JVM-based services with large heap sizes * You want to reduce node count for easier management (fewer nodes to patch, drain, monitor) * Your workload has high pod-to-pod communication — co-locating more pods on fewer, larger nodes reduces network hops * Spot/preemptible node costs make larger nodes more cost-effective per vCPU **The production answer — use both:** Most mature clusters use multiple node groups: a standard node group for general workloads scaled horizontally, and a high-memory or GPU node group for specialized workloads. Kubernetes node selectors and taints route workloads to the appropriate node type. ```yaml ## Node selector — this pod only schedules on memory-optimized nodes spec: nodeSelector: node.kubernetes.io/instance-type: r5.2xlarge ``` --- ### 30. Your organization is moving from a monolithic deployment to microservices. You are asked to design the CI/CD strategy for 20 new services. What decisions do you make upfront and why? **What the interviewer is testing:** Architecture-level thinking about CI/CD at scale. This is a design question — there is no single correct answer, but weak answers ignore the tradeoffs entirely. **Answer:** Getting CI/CD architecture right for 20 services from the start is far cheaper than refactoring 20 independent pipelines later. The decisions that matter most upfront: **Decision 1 — Monorepo or polyrepo** Monorepo (all 20 services in one Git repo) gives you: atomic commits across services, shared tooling and conventions, easier dependency management between services. The cost: CI must be smart about path-based builds or every commit rebuilds all 20 services. Polyrepo (each service in its own repo) gives you: clear ownership, independent versioning, no accidental coupling. The cost: cross-service changes require multiple PRs, shared tooling must be distributed, harder to enforce standards. For a new migration, monorepo with path-based CI is usually the better starting point — you can always split later, but merging is painful. **Decision 2 — Shared pipeline templates** Write one pipeline template that all 20 services reuse. Individual services override only what is different. ```yaml ## GitHub Actions reusable workflow — one definition, 20 callers ## .github/workflows/service-pipeline.yml (template) on: workflow_call: inputs: service_name: required: true type: string dockerfile_path: default: "./Dockerfile" type: string ## Each service calls it with 3 lines: ## uses: ./.github/workflows/service-pipeline.yml ## with: ## service_name: payment-service ``` If each of 20 services has its own independent pipeline YAML, you have 20 places to fix when a security scanner version needs updating. With templates, you fix one file. **Decision 3 — Shared image registry with consistent tagging** All services push to the same ECR or Docker Hub org. Every image tagged with `git-sha`, `branch-name`, and `latest` on main. This makes traceability trivial — which commit is running in production is always answerable. **Decision 4 — Deployment via GitOps from day one** Start with a GitOps deployment model (ArgoCD) before you have 20 services, not after. Retrofitting GitOps onto 20 independent deploy scripts is a multi-month project. Starting with it means all 20 services get consistent deployment, rollback, and drift detection automatically. **Decision 5 — Shared observability standards** Every service must expose `/health/live`, `/health/ready`, and `/metrics` before it gets a pipeline. This is not optional — it is the entry ticket to the CI/CD system. Services that do not instrument themselves do not deploy to production. **What you say in the interview:** "The most expensive CI/CD mistake is treating each service as a unique snowflake. The value of microservices is independent deployability, not independent pipelines. Shared templates and standards are what make 20 services manageable by a 3-person platform team." ---
### 31. It is Thursday afternoon. Your team is about to do a large deployment — a rewrite of the authentication service that touches every other service. Your engineering manager asks your recommendation: deploy today or wait until Monday? **What the interviewer is testing:** Risk judgment and communication. Technical competence combined with business awareness. There is no single correct answer — they are evaluating how you think about it. Your team is about to do a large deployment — a rewrite of the authentication service that touches every other service. Your engineering manager asks your recommendation: deploy today or wait until Monday? **What the interviewer is testing:** Risk judgment and communication. Technical competence combined with business awareness. There is no single correct answer — they are evaluating how you think about it. **Answer:** Deploy Monday. And here is exactly how you explain it: A Thursday afternoon deployment of a critical, cross-cutting change has a high blast radius and a short recovery window. If it fails: * It is 4 PM Thursday — your team is available, but only for a few more hours * If it takes more than 3-4 hours to stabilize, you are debugging at 8 PM with a tired team * If it requires data recovery or is not resolved by end of day, you are looking at a weekend incident Wait until Monday and you get: * Full team availability for the entire day * A rollback window of 8 hours before European timezone users go offline * The ability to do a proper canary — 10% of traffic for 2 hours — before full rollout * Your on-call engineer is alert, not tired from a week of work **What you say to the manager:** "I recommend Monday. Not because the code is not ready, but because the risk profile of a late-week deployment of an auth rewrite is bad. If something goes wrong today, we debug it tired on a Friday evening or the weekend. Monday gives us full-day coverage. If the business need requires it today, I want us to have explicit rollback criteria — if error rate on any dependent service crosses X% within 10 minutes, we auto-revert without discussion. Can we agree on that threshold now?" This answer shows: risk thinking, clear communication, not just saying "no" but offering a concrete path forward. --- ### 32. You join a team that has no monitoring, no alerting, and finds out about production problems from customers. You have 4 weeks to improve the situation. What do you do in what order? **What the interviewer is testing:** Prioritization under real constraints. You cannot do everything in 4 weeks. **Answer:** Week 1 — Basic availability visibility. Install Prometheus and Grafana. Add the standard Kubernetes dashboards (cluster CPU, memory, pod restarts). Add the node exporter. Set up one alert: pod restarts more than 3 times in 5 minutes sends a Slack message. This is not perfect but it is dramatically better than nothing. Cost: one engineer, 2-3 days. Week 2 — Service-level error rates. Instrument the top 3 most critical services (the ones customers contact you about most often) with a RED dashboard — rate, errors, duration. Add one alert per service: error rate above 1% for 5 minutes. This is when you stop hearing about problems from customers first. Week 3 — Logging. Ship application logs to a central location (Loki or CloudWatch Logs). Make sure every log line has a correlation ID so you can trace a single request across services. Add a Grafana dashboard that shows logs alongside the metrics from Week 2. Week 4 — On-call process. Write a runbook for the top 5 most common alert types — what the alert means, the first 3 commands to run, and who to escalate to. Test the on-call workflow once before it goes live. **What you say explicitly in the interview:** "I would not try to build a perfect observability stack in 4 weeks. I would pick the changes with the highest signal-to-noise improvement and implement those. A Prometheus alert that fires on pod CrashLoopBackOff is worth more than a beautifully architected ELK stack that takes 3 weeks to deploy." --- ### 33. Your Kubernetes cluster is running 95 nodes. You need to upgrade the Kubernetes version from 1.27 to 1.29. How do you do it with zero application downtime? **What the interviewer is testing:** Real cluster operations knowledge. Zero-downtime upgrades are possible but require understanding PDBs, node draining, and rolling upgrades. **Answer:** This is a multi-step process. The order matters. **Pre-upgrade checklist:** ```bash ## 1. Check all APIs used in your manifests are still available in 1.29 ## Some APIs are deprecated and removed between versions kubectl api-versions ## Check the Kubernetes deprecation guide for 1.27 → 1.29 breaking changes ## 2. Test the upgrade in staging first ## Run your full deployment on a staging cluster at 1.29 ## 3. Verify all pods have appropriate PodDisruptionBudgets kubectl get pdb --all-namespaces ## Any critical service without a PDB will have downtime risk during node drains ``` **Upgrade sequence — control plane first, workers second:** Managed Kubernetes (EKS, GKE, AKE) handles the control plane upgrade for you — you change the version and the cloud provider rolls it over. ```bash ## EKS example — upgrade control plane aws eks update-cluster-version \ --name production-cluster \ --kubernetes-version 1.29 ## Wait for this to complete before touching nodes ``` **Node upgrade — rolling, one node group at a time:** ```bash ## Create a new node group at 1.29 (EKS managed node groups) ## Old node group stays running while new one provisions ## Cordon the first old node (mark it unschedulable for new pods) kubectl cordon old-node-1 ## Drain it — move existing pods to healthy nodes ## --ignore-daemonsets: daemon sets redeploy automatically ## --delete-emptydir-data: pods using emptyDir storage will lose it kubectl drain old-node-1 \ --ignore-daemonsets \ --delete-emptydir-data \ --grace-period=120 ## The drain respects your PodDisruptionBudgets ## If draining would violate a PDB, it waits until a replacement is running elsewhere ## Once drained, terminate the node ``` Repeat for each old node, one at a time. Kubernetes reschedules pods from drained nodes onto available nodes in the new node group running 1.29. **Zero downtime depends on:** * Every critical service having a PDB that keeps at least one instance running * Services having more than one replica (you cannot drain safely with single replicas) * The new nodes having enough capacity to absorb the pods from the nodes being drained --- ### 34. You discover that developers on your team have been directly `kubectl exec`-ing into production pods to debug issues and sometimes making changes. How do you address this? **What the interviewer is testing:** Security posture and culture simultaneously. This is a real problem at almost every company. **Answer:** First, acknowledge why it happens before trying to stop it. Engineers `exec` into pods because it is the fastest way to debug a live problem. If you remove the capability without providing a better debugging workflow, you create friction without fixing the root cause — engineers will find another workaround. **The immediate security response:** ```yaml ## RBAC — remove exec permission from the default developer role apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: developer rules: - apiGroups: [""] resources: ["pods", "pods/log"] verbs: ["get", "list", "watch"] ## can view pods and logs ## No "pods/exec" resource here — cannot exec into pods ``` **But also audit what happened:** ```bash ## Kubernetes API audit logs capture every exec ## Find all exec calls in the last 7 days kubectl get events | grep exec ## Or query CloudWatch/Cloud Logging if your cluster ships audit logs there ``` Were any changes made? If someone exec'd in and modified a config file or ran a one-off command that changed behavior, that change is not in version control — it is invisible drift. **The better debugging alternative — ephemeral debug containers:** ```bash ## Kubernetes 1.23+ supports ephemeral containers ## Attach a debugging container to a running pod without modifying the pod spec kubectl debug -it <pod-name> \ --image=nicolaka/netshoot \ ## debugging toolkit with network tools --target=<container-name> ## share the process namespace ## This is audited, temporary, and does not allow modifying the application ``` **The cultural response:** Have a direct conversation with the team: "I understand why exec feels faster. Here is the problem — changes made inside pods are invisible, unaudited, and wiped when the pod restarts. We have had a bug stay in production for 3 days because someone fixed it with exec and it disappeared in the next deployment without anyone knowing." Offer: "I am going to build a debugging runbook for our top 5 most common issues that does not require exec. If there is a scenario we cannot debug without exec, tell me and I will fix the observability gap." This answer shows security knowledge plus the cultural intelligence to address the root cause rather than just adding a lock. --- ### 35. A critical microservice is consuming 90% of your monthly AWS bill due to NAT Gateway data transfer costs. Your manager asks you to cut it by 60% without changing the application code. What do you investigate and what do you change? **What the interviewer is testing:** Cloud cost optimization with real constraints. "Without changing application code" is the key — you need infrastructure solutions only. **Answer:** NAT Gateway costs in AWS have two components: hourly charge (~$0.045/hour per AZ) and data processing charge ($0.045/GB). At scale, the data processing charge is almost always the dominant cost. Before optimizing, understand what traffic is flowing through NAT Gateway. **Step 1 — Enable VPC Flow Logs and identify the traffic** ```bash ## Enable flow logs to S3 or CloudWatch aws ec2 create-flow-logs \ --resource-type VPC \ --resource-ids vpc-xxxxx \ --traffic-type ALL \ --log-destination-type s3 \ --log-destination arn:aws:s3:::vpc-flow-logs-bucket ## Query with Athena after 30 minutes to see top destinations by byte count ``` **The most common causes and their fixes:** *EC2 or pods downloading from S3 via NAT Gateway.* Fix: create a VPC Endpoint for S3. It is free, keeps traffic inside AWS, and avoids NAT Gateway entirely. ```bash aws ec2 create-vpc-endpoint \ --vpc-id vpc-xxxxx \ --service-name com.amazonaws.ap-south-1.s3 \ --route-table-ids rtb-xxxxx ## After this, S3 traffic routes through the endpoint, not NAT Gateway ## Zero cost for S3 data transfer via endpoint ``` *Pods calling AWS API endpoints (ECR, CloudWatch, Secrets Manager, STS) via NAT Gateway.* Fix: create Interface VPC Endpoints for each AWS service used. *Cross-AZ data transfer amplified by NAT Gateway.* If your pods are in AZ-a but the NAT Gateway is in AZ-b, every outbound request crosses AZs twice — to the NAT Gateway and back. Fix: ensure one NAT Gateway per AZ and route each AZ's private subnets to their local NAT Gateway. *Unnecessary internet calls from the application.* Check flow logs for traffic to destinations outside AWS. CDN assets being fetched server-side, dependency update checks, telemetry calls — none of these need to go through NAT Gateway if you add a caching proxy or disable them. A combination of S3 VPC endpoint plus interface endpoints for ECR and CloudWatch typically cuts NAT Gateway costs by 50-80% for containerized workloads without touching a single line of application code. --- ### 36. Your team wants to implement feature flags. A developer asks you to add a `FEATURE_NEW_CHECKOUT=true` environment variable to the Kubernetes deployment. What is wrong with this approach and what do you propose instead? **What the interviewer is testing:** Platform thinking. Feature flags via environment variables is a common antipattern that creates operational problems. **Answer:** The fundamental problem: environment variables in a Kubernetes Deployment require a pod restart to change. Toggling a feature flag via an env var means restarting pods every time you want to enable or disable a feature. In production this means a rolling deployment, which takes minutes and temporarily reduces capacity. The secondary problem: you cannot change env vars for a percentage of traffic. If you want to roll out a feature to 10% of users, env vars cannot do that — they apply to the entire deployment. **What breaks at scale:** * You have 20 features in various states of rollout. Your Deployment has 20 `FEATURE_X=true/false` environment variables. Changing any one of them triggers a full rolling restart. Your on-call engineer gets an alert every time a feature flag is toggled. * If the new checkout feature causes an error, your rollback is another full rolling restart — not an instant toggle. **The right approach — a feature flag service:** Use a dedicated feature flag system: LaunchDarkly, Flagsmith (self-hosted), Unleash, or AWS AppConfig. ```python ## Application reads flag at runtime from the service, not at startup from env import ldclient ldclient.set_sdk_key("sdk-key") client = ldclient.get() def handle_checkout(user): ## Flag evaluated per-request — no restart needed to change it if client.variation("new-checkout-flow", {"key": user.id}, False): return new_checkout_handler(user) else: return old_checkout_handler(user) ``` This gives you: instant toggles with no restarts, percentage rollouts (10% of users see the new flow), user targeting (enable for internal users only), and kill switches that work in under a second. **If a full feature flag service is too much overhead right now:** Use Kubernetes ConfigMaps with a mounted volume (not env vars). ConfigMap volume mounts update in running pods without a restart — the file changes in place and the application can watch for changes. ```yaml ## ConfigMap change propagates to pods without restart ## (applications must read the file, not just on startup) volumeMounts: - name: feature-flags mountPath: /config/flags volumes: - name: feature-flags configMap: name: feature-flags ``` **What to say in the interview:** "The env var approach is the kind of solution that works fine for one team on one service but does not scale. When I see `FEATURE_X` environment variables in a production Deployment, I ask how many restarts happen per week because of flag changes — and that number usually makes the case for a proper solution." --- ### 37. Production is down. You are the on-call engineer. The CEO is on a call with a major client and your VP of Engineering sends you a Slack message: "Fix this NOW." You have been investigating for 8 minutes and have not found the root cause yet. What do you do? **What the interviewer is testing:** Handling pressure from leadership during an incident. This is a communication and incident management question, not a technical one. **Answer:** This is a test of whether pressure from above causes you to take shortcuts that make things worse. **What you do not do:** Stop your systematic investigation to give the VP a satisfying response. Rushing your diagnosis leads to wrong guesses, hasty changes that worsen the incident, and a longer total downtime. "Fix it NOW" applied to a half-diagnosed problem usually extends the outage. **What you do — respond immediately but briefly:** Reply to the VP within 60 seconds: > "Acknowledged. Investigating actively. Current finding: [one sentence on what you know so far]. Will update every 5 minutes. Not deploying any changes until root cause is identified." This does three things: shows you are on it, gives them something concrete, and sets a cadence so they do not need to chase you. **Continue the technical investigation without interruption.** If you have a hypothesis after 3 more minutes, share it: > "Update: looks like a database connection pool issue. Checking connection counts now. If confirmed, fix is a connection limit increase — 2 minutes to apply." **What if the VP escalates to your manager?** Your manager should shield you from interruption during active diagnosis. If they cannot, and someone is calling you: "I need 2 minutes uninterrupted to test this hypothesis. I will call you back immediately." Then do it. **The professional response to "Fix it NOW":** "I understand the urgency. The fastest path to resolution is an accurate diagnosis — guessing and making the wrong change adds time. I am 8 minutes in and narrowing it down. You will have an update from me in 4 minutes with either the fix or a clear picture of what I still need to determine." **What this answer signals in an interview:** You do not panic under management pressure. You communicate proactively. You understand that rushing a diagnosis is usually slower than doing it right. These are the behaviours that get engineers trusted with production systems. --- ### 38. You are asked to reduce your team's deployment frequency from once a week to multiple times per day. The CTO says this is the priority. Where do you start and what will slow you down? **What the interviewer is testing:** Continuous delivery maturity thinking. Moving from weekly to daily deploys requires more than just pipeline changes. **Answer:** Weekly to multiple-times-daily is a significant shift. The pipeline is usually not the bottleneck. Start by understanding what prevents daily deploys now. **Diagnose the current constraint first:** Ask three questions: Why is the team deploying weekly? What would break if you deployed today's commit right now? What takes the longest between code complete and production? Common answers: manual approval gates, long test suites, manual QA sign-off, environment promotion queues, fear of breaking production. **The changes that actually enable higher frequency — in order of impact:** **1. Automated rollback and fast detection.** Deploying more often means more chances for a bad deploy. Before increasing frequency, you must be able to detect a bad deploy within 2 minutes and roll it back within 5. Without this, higher frequency = more incidents. **2. Feature flags instead of feature branches.** Weekly deploys often exist because features are not complete — they sit in long-lived branches waiting. Feature flags let you deploy incomplete features safely: they are deployed but disabled until ready. This removes the "wait for feature complete" release cycle. **3. Automated testing that engineers trust.** If the test suite has false positives, engineers wait for manual validation to override them. Fix the flaky tests before increasing deploy frequency — flaky gates are why people route around gates. **4. Small, reversible changes.** A weekly deploy accumulates a week of changes. One bad commit is buried in 50 commits. A daily deploy means 5-10 commits. Finding the culprit after a bad deploy is a `git bisect` away. Smaller batches make problems easier to locate and roll back. **What will slow you down:** * Database migrations that are not backward compatible — you cannot deploy twice in one day if each deploy requires a breaking schema change * Manual QA processes that cannot run more than once per week * Monolithic deployment that deploys everything even when only one service changed * Lack of staging environment that resembles production closely enough to catch issues **What to say in the interview:** "I would not go from weekly to multiple times daily in one step. I would target daily first, run it for a month, fix the friction points, then push toward multiple times daily. Skipping steps creates incidents that set the programme back." --- ### 39. A new engineer on your team keeps manually editing Kubernetes resources in production using `kubectl edit` instead of updating the manifests in Git. The changes work but they cause confusion when the GitOps operator reconciles them away. How do you address this? **What the interviewer is testing:** Cultural and process problem-solving. This is not a technical question — it is about how you change behaviour without creating conflict. **Answer:** Before addressing the behaviour, understand why it is happening. Engineers do not bypass version control to be difficult — they do it because it is faster for their immediate need, or because they do not understand the consequences, or because the correct path has too much friction. **The conversation first:** Talk to the engineer directly, not in public, and not with blame: > "I noticed some production resources were edited directly. Can you walk me through what you were trying to do? I want to make sure the right workflow is clear and not creating unnecessary friction for you." This usually surfaces one of: they did not know ArgoCD would revert it, they needed to test something quickly and did not know a faster path through Git, or the PR/review process is too slow for urgent production fixes. **Fix the root cause, not the symptom:** If the problem is "I needed to test something quickly": set up a dev namespace where manual changes are allowed. Production stays GitOps-controlled. If the problem is "the PR process is too slow for urgent fixes": create a fast-path process — a PR to the gitops repo requires only one approval and merges in under 10 minutes for production hotfixes. If the problem is "I did not know ArgoCD reverts manual changes": this is a knowledge gap, not a bad actor. A 15-minute walkthrough of how ArgoCD works and why the rule exists usually resolves it permanently. **The technical reinforcement — after the conversation:** Once the engineer understands and agrees, add a monitoring alert for manual drift: ```bash ## ArgoCD can alert on OutOfSync apps — configure in AlertManager ## This makes drift visible without blocking work ``` If the behaviour continues after the conversation and education, escalate to RBAC restriction — remove write access to production namespaces from developer accounts: ```yaml ## Developers can read but not modify production resources kind: Role rules: - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch"] ## no update, patch, or delete ``` **What to say in the interview:** "I would not start with RBAC restriction. Locking down access before explaining why creates resentment and does not change the mental model. The sequence is: understand the root cause, fix the workflow friction, explain the consequence of bypassing it, then enforce technically if the behaviour persists." --- ### 40. Your team's staging environment is consistently 3-4 weeks behind production in terms of configuration, secrets, and data. Developers are finding bugs in production that staging never caught. How do you fix this? **What the interviewer is testing:** Environment parity — a classic engineering operations problem that almost every team has dealt with. **Answer:** Staging catching less than production is a symptom, not the problem. The problem is configuration drift between environments. Find every place where staging and production diverge, and systematically eliminate each divergence. **Audit the current differences:** ```bash ## Compare Kubernetes resources between namespaces kubectl get configmaps -n staging -o yaml > staging-configs.yaml kubectl get configmaps -n production -o yaml > prod-configs.yaml diff staging-configs.yaml prod-configs.yaml ## Compare environment variables across deployments kubectl get deployments -n staging -o json | jq '.items[].spec.template.spec.containers[].env' ## Do the same for production and diff ``` **The common causes of environment drift — and their fixes:** *Secrets are different.* Production has real credentials, staging has test credentials — this is expected. But if staging is missing secrets entirely that production has, those code paths never get tested. Fix: ensure staging has equivalents of every secret that exists in production (pointing to test/sandbox versions of each dependency). *Infrastructure sizes are different (fine) but behaviours are different (not fine).* Staging might have 1 replica vs production's 10. That is okay. But if staging has a different memory limit that prevents an OOM condition that production hits, that is a problem. Use the same resource ratios. *Manual production changes that never made it back to Git.* This is the most insidious cause. Someone `kubectl edit`-ed a ConfigMap in production three weeks ago to fix an incident. Staging never got that change. Fix: implement the GitOps control from Q39 — if all production changes go through Git, staging can always be synchronized from the same Git source. *Third-party service versions differ.* Production has Redis 7.0, staging has Redis 6.2. Some Redis 7.0 features are used in code. Staging never tests them. Fix: pin third-party service versions in your Helm values and use the same version file for both environments. **The structural fix:** Use the same Terraform modules and Helm charts for both environments, with environment-specific values files only for things that genuinely need to differ (instance sizes, replica counts, endpoint URLs): ``` charts/payment-service/ values-staging.yaml ← overrides for staging (smaller instances, test endpoints) values-production.yaml ← overrides for production values-common.yaml ← everything that must be identical between environments ``` **What to say in the interview:** "The goal is not identical environments — it is equivalent behaviour. Staging should exercise every code path that production exercises. If a code path only runs in production, that path is untested. I would treat environment parity as a reliability metric: what percentage of production behaviour is reproducible in staging?" ---
### 41. Tell me about a time you introduced a change that caused a production incident. What happened, and what did you change afterward? **What the interviewer is testing:** Self-awareness, accountability, and whether you actually learn from mistakes. Every experienced engineer has caused an incident. The answer reveals whether you treat it as a learning event or a thing to minimize. **What a strong answer looks like:** Be specific. Vague answers ("I once made a config change that caused some issues") signal that you either do not remember clearly or are hiding details. Name the service, the change, the impact, the timeline. Interviewers have seen enough post-mortems to know when an answer is being sanitized. The structure: what you changed and why, what broke and when you found out, how long it took to resolve and why, what you changed in your process specifically afterward. The "what you changed afterward" is what they are actually grading. An answer that ends at "we fixed it" signals you did not learn anything concrete. An answer that ends with "I added a canary deployment step to that pipeline which catches this class of issue before it reaches 100% of traffic" signals that you systematically reduced the risk of recurrence. **What kills the answer:** Blaming someone else, saying the incident was not really your fault, or saying you have never caused an incident (nobody believes this and it signals low ownership). --- ### 42. Tell me about a time you disagreed with a technical decision made by someone senior to you. What did you do? **What the interviewer is testing:** Whether you can push back constructively without being passive-aggressive or going around people. Companies want engineers who speak up when they see a problem — not ones who silently comply and then say "I told you so" later. **What a strong answer looks like:** Describe the disagreement specifically — what the decision was and what your concern was. Describe how you raised it: in a private conversation first, with evidence, not in a public meeting designed to embarrass. Describe the outcome — either you were persuaded by their reasoning you had not considered, or you persuaded them, or you disagreed and committed to the decision anyway. The last option is the most mature: disagreeing and committing. You raised your concern clearly, the decision was made, and you executed it fully without sabotaging it. This is what senior engineers actually do. **What kills the answer:** Going around the person to their manager, passive resistance, or framing the story so you were clearly right and they were clearly wrong. Real technical disagreements are rarely that clean. --- ### 43. Describe a project where you had to learn a technology you had never used before under time pressure. How did you approach it? **What the interviewer is testing:** Learning agility and how you handle gaps in knowledge. Mid-level interviews frequently surface new tooling — ArgoCD if you have not used it, Terraform if your background is Ansible, GCP if you have only done AWS. **What a strong answer looks like:** Describe specifically how you learned: documentation first or tutorials first, whether you built a proof of concept before integrating, what questions you asked and who you asked them to, how long before you felt productive. The interviewer cares less about what you learned and more about whether you have a repeatable method for learning. Engineers who say "I just Googled it" have the same method as everyone else. Engineers who say "I find the official getting-started guide, run the simplest possible working example, then read the architecture overview to understand why the decisions were made that way" have a method. --- ### 44. Tell me about a time you improved a process that was causing friction for your team. How did you identify the problem and what did you change? **What the interviewer is testing:** Proactive ownership. Mid-level engineers are expected to notice operational friction and fix it — not just complete assigned tasks. **What a strong answer looks like:** The identification story matters. Did you notice it because you were directly affected, because a teammate complained, or because you looked at a metric (pipeline times, deployment frequency, MTTR)? Engineers who spot problems through measurement signal more maturity than ones who only notice problems when they personally feel them. Describe the specific change you made and the specific improvement it produced. "I optimized the pipeline" is weak. "I added dependency caching to the CI pipeline which cut average build time from 18 minutes to 6 minutes, which reduced the complaint rate about slow CI by removing the main reason developers were skipping tests locally" is strong. --- ### 45. Tell me about a time you had to work with a developer team that did not follow DevOps practices — no tests, manual deployments, no version control discipline. How did you handle it? **What the interviewer is testing:** Cross-functional collaboration and influence without authority. DevOps engineers routinely work with teams who do not share the same practices. **What a strong answer looks like:** The wrong approach: issuing mandates, going to management, or refusing to help until they fix their practices. These create resentment and do not change behaviour. The right approach: start with understanding why they work the way they do. A team with no tests usually has no tests because nobody ever made time to write them, not because they are opposed to testing. A team with manual deployments usually has them because automating felt risky and nobody had the time to invest in making it safe. Describe one specific practice you changed, how you changed it, and the team's reaction. Did you write the first tests yourself to show them the pattern? Did you automate one deployment path to demonstrate that it was safer than manual? Concrete change, concrete outcome. --- ### 46. Describe a time you had to give difficult feedback to a teammate about something technical — code quality, unsafe practices, or a pattern that was creating problems for the team. **What the interviewer is testing:** Interpersonal maturity and the ability to have difficult conversations. Mid-level engineers are expected to mentor juniors and flag issues — not stay quiet to avoid conflict. **What a strong answer looks like:** Describe the issue specifically. Describe how you raised it — in private, framed around the impact to the system rather than criticism of the person, with a specific suggested improvement rather than just identifying the problem. The outcome matters less than the approach. You can give excellent feedback that the person does not immediately act on — that is not a failure. The failure is staying quiet because it is uncomfortable. One thing to say explicitly: "I made sure I understood their reasoning first before suggesting a change. There are patterns that look wrong that have a history behind them — I wanted to know if there was context I was missing before treating it as a mistake." --- ### 47. Tell me about a time you had to make a decision with incomplete information during a production incident. What did you decide and how did you decide it? **What the interviewer is testing:** Decision-making under uncertainty. Production incidents never give you perfect information. Waiting for certainty before acting is itself a decision — often the wrong one. **What a strong answer looks like:** Describe the incident and what information you had at the decision point. Describe the options you considered and what made you choose the action you chose — specifically, what was your confidence level and what was the reversibility of the action. The best framework to articulate: "I did not know the root cause with certainty, but I had enough evidence to be 70% confident this was the cause. The rollback was reversible in under 3 minutes if I was wrong. The cost of being wrong was 3 extra minutes of downtime. The cost of not rolling back if I was right was 20+ more minutes of downtime. So I rolled back." Reversibility is the key variable in uncertain production decisions. Actions that are easy to reverse (rollbacks, scaling up) are safe to take with incomplete information. Actions that are hard to reverse (dropping a database, migrating data) require higher certainty. --- ### 48. Tell me about a situation where you pushed back on a request from a stakeholder that you believed was technically unsafe or created unacceptable risk. How did you handle it? **What the interviewer is testing:** Whether you treat security and reliability as non-negotiable or as preferences to be traded away under business pressure. **What a strong answer looks like:** Describe the request, the specific risk you identified, and how you communicated it. The important thing: you did not just say "no." You explained the risk in terms the stakeholder could understand (impact on customers, regulatory risk, financial exposure) and you proposed an alternative that met the business need more safely. "I pushed back on the request to store user passwords in plain text in the database. I explained that this would expose us to significant regulatory liability under GDPR and create a reputational risk we could not recover from quickly. I proposed using bcrypt hashing, explained that it would take an extra day of work, and offered to write the implementation myself to remove the time concern from the decision." **What kills the answer:** Framing it as you being right and the stakeholder being stupid. Or describing a situation where you said no and nothing happened. The test is whether you engaged constructively. --- ### 49. Describe a time when you had to hand off a project or system to another engineer. What did you do to make the handoff successful? **What the interviewer is testing:** Documentation discipline and team thinking. Engineers who do not document well create single points of failure. Mid-level engineers should be thinking about what happens when they leave or move to a different project. **What a strong answer looks like:** Describe what you produced for the handoff: runbooks, architecture diagrams, on-call procedures, known quirks of the system, the things you knew that were not written down anywhere. The specificity here matters — "I wrote documentation" is weak, "I wrote a runbook covering the five most common production issues and their resolution steps, recorded a walkthrough video of the deployment process, and paired with the receiving engineer for two weeks before fully handing off" is strong. Also describe what you learned from the handoff about your own documentation gaps. "During the pairing weeks I discovered three things I knew instinctively but had not written down. I added them to the runbook and used that as a template for documenting the next system I owned." --- ### 50. Where do you see the difference between a mid-level and senior DevOps engineer? What are you working on to close that gap? **What the interviewer is testing:** Self-awareness about your own growth and whether you are actively developing or waiting for a promotion to tell you what to learn. **What a strong answer looks like:** Name specific, concrete differences — not vague statements like "seniors have more experience." The real differences: seniors design systems others build on, they identify problems before they are asked to, they have enough context to push back on requirements rather than just implementing them, they think about organizational change not just technical change. Then name what you are specifically working on. "I am working on my ability to drive architectural decisions in group settings — I find I can identify the right technical direction but I lose confidence when I need to defend it against more senior engineers in a room. I have been deliberately volunteering to lead design reviews to build that muscle." This level of self-awareness is exactly what distinguishes candidates who get the mid-to-senior transition from those who stay mid-level for years. **What kills the answer:** Saying the difference is just experience or seniority level without substance. Or not having anything specific you are working on — that signals you are waiting for growth to happen to you rather than seeking it. ---
**Mid-Level DevOps Engineer (2-5 years)** | Company Type | Range | |:---|:---| | Service / IT company | ₹10L - ₹18L | | Mid-size product startup | ₹16L - ₹24L | | Well-funded growth startup | ₹20L - ₹30L | | Large product company (Atlassian, Stripe, PhonePe tier) | ₹24L - ₹38L | | FAANG / hyperscaler India offices | ₹30L - ₹50L+ | These numbers assume you can answer Tier 2 questions with real examples from production systems you have owned — not just described in theory. Candidates who answer every question with "I read about this" without hands-on context land at the lower end or do not proceed past the technical screen. The gap between ₹18L and ₹35L for the same years of experience is almost entirely explained by depth of ownership — whether you built and ran systems under production pressure, or whether you contributed to systems someone else owned and operated.
You cleared the Junior round. You can explain what a pod is, you know how Docker layers work, and you can write a basic ...
> If you need to look any of these up, go back to the relevant module first. These are the floor, not the ceiling. Kuber...
CI/CD — Architecture and Decisions 1. Your team has a monorepo with 12 microservices. The CI pipeline builds and tests a...
31. It is Thursday afternoon. Your team is about to do a large deployment — a rewrite of the authentication service that...
41. Tell me about a time you introduced a change that caused a production incident. What happened, and what did you chan...
Mid-Level DevOps Engineer (2-5 years) Company Type Range Service / IT company ₹10L - ₹18L Mid-size product startup ₹16L ...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.