- 5-8 years experience - Roles: Senior DevOps Engineer, Staff Engineer Infrastructure, Platform Engineering Lead - 63 checklist questions, 45 real interview Q&A, 10 live scenarios, 20 behavioral questions - Companies: Atlassian, PhonePe, Cloudflare, Zerodha, Razorpay, Databricks
You have been in production. You have debugged things at 2 AM. You have written Terraform, built pipelines, and owned a Kubernetes cluster. That is not what this round tests. Senior interviews test a different set of things entirely. The questions shift from "can you do this" to "how do you decide whether to do this at all." You will be asked to design systems you have never built, make cost tradeoffs between options that both work, lead an incident with ten engineers watching you, and explain to an engineering manager why a six-month migration is worth doing. Three things separate senior candidates from strong mid-level candidates. You reason about tradeoffs out loud without being asked — you give the answer and immediately explain what it costs and when you would choose differently. You treat the human and organisational problem as part of the technical problem — a migration plan with no change management is not a real plan. You use numbers — "reduce deployment frequency from twice a week to six times a day" is a senior answer, "it will be faster" is not. This module has four sections. Every question across Tier 2, Tier 3, and the Behavioral Round is numbered continuously so you always know exactly where you are. **Tier 1 — Senior Fundamentals Checklist (no answers)** 63 questions across 5 categories: Kubernetes platform internals, Infrastructure and Platform, Observability and SRE, Security and Compliance, and Cost and Platform Strategy. These are table stakes for a senior round. If you cannot answer these cold, go back to the relevant module first. No answers provided — use this as a readiness check before Tier 2. **Tier 2 — Real Interview Questions (Q1 to Q45)** 45 questions asked at Atlassian, PhonePe, Databricks, Cloudflare, Stripe, Zerodha, Razorpay, and large Indian product companies at senior and staff level. Covers system design and architecture, CI/CD at scale, Kubernetes operations, SRE and observability, security, cost optimisation, platform strategy, and team leadership. Every answer includes real code, commands, and the tradeoffs a senior engineer is expected to articulate. **Tier 3 — Scenario Round (Q46 to Q55)** 10 open-ended senior scenarios — no single correct answer exists. The interviewer is watching how you think, communicate risk, and make decisions with incomplete information. Covers: joining a company with no DevOps maturity, architecture decision records, AWS cost spikes, on-call burnout, security audit remediation, post-mortem resistance, inherited legacy systems, platform migration proposals, deployment risk judgment, and build vs buy decisions under pressure. **Behavioral Round (Q56 to Q75)** 20 behavioral questions with full answers covering every signal senior interviews test: production incidents you caused, pushing back on stakeholders, influencing without authority, public disagreements, peer feedback, delivering under resource constraints, cross-team conflict, managing up, decisions you would make differently, incident command across teams, killing your own projects, escalation judgment, and the gap between senior and staff engineer level.
These are table stakes. If you need to look any of these up, go back to the relevant module first. ### Kubernetes - Platform Level * What is the Kubernetes control plane and what does each component do? * How does etcd work and what happens if it goes down? * What is a CRD and why would you create one? * What is the Cluster Autoscaler and how does it interact with HPA? * What is KEDA and when would you use it over standard HPA? * How does Kubernetes handle node failure — what happens to pods on a dead node? * What is a PodTopologySpreadConstraint and why does it matter for availability? * What is the difference between a soft and hard affinity rule? * How does the Kubernetes garbage collector work? * What is a validating and mutating admission webhook? ### Infrastructure and Platform * What is the difference between infrastructure drift and configuration drift? * How does Terraform remote state locking work and why does it matter? * What is a Terraform provider and how do you write a custom one? * What is the difference between blue/green and canary at the infrastructure level vs application level? * What is a service account token and how does IRSA replace it on EKS? * What is AWS Service Control Policy and how does it differ from IAM? * What is a transit gateway and when would you use it over VPC peering? * What is AWS PrivateLink and what problem does it solve? ### Observability and SRE * What is multi-window multi-burn-rate alerting and why is it better than single-threshold alerts? * What is an error budget policy and how does it change team behaviour? * What is exemplar data in Prometheus and why does it matter for correlation? * What is the difference between black-box and white-box monitoring? * What is chaos engineering and how do you run it safely in production? * What is MTBF vs MTTR and which one should you optimise for first? ### Security and Compliance * What is RBAC vs ABAC and when would you use attribute-based access control? * What is a supply chain attack and how do you defend against it at the build pipeline level? * What is SBOM and why are organisations starting to require it? * What is the difference between encryption at rest and encryption in transit? What protects data in use? * What is a CVE and how do you triage which ones to patch first? ### Cost and Platform Strategy * What is FinOps and what does a senior DevOps engineer own in a FinOps practice? * What is the difference between a reserved instance, a savings plan, and spot? * What is rightsizing and how do you automate it? * What is a platform team and how is it different from a shared services team? * What is an internal developer platform and what problem does it solve?
### System Design and Architecture ### Designing Scalable Infrastructure from First Principles ### The Question You are joining a funded fintech startup with 15 engineers and no existing infrastructure. They are launching in three months. How do you design and build their production infrastructure? ### What the interviewer is testing This is not a trick question — there is no single correct answer. They are testing whether you can make time-bound, risk-aware architectural decisions rather than defaulting to maximum complexity because you know how to build it. The worst answer: immediately designing a multi-region, service-mesh-enabled, full observability stack with separate platform and application teams. That is correct for a company at scale. It is wrong for a three-month launch with 15 engineers. ### The right approach Start by establishing constraints. Three months to launch, 15 engineers, fintech — that means regulatory considerations (PCI-DSS if they are processing payments, RBI guidelines if they are an Indian NBFC), and the need to move fast without burning the team on infrastructure before the product is proven. The architecture should be simple enough that the engineering team can own it without a dedicated SRE. That means managed services over self-managed wherever the cost is reasonable. Internet | v AWS ALB (Application Load Balancer) | v ECS Fargate (containerised services, no server management) | +-----------+ | | v v RDS Aurora ElastiCache Redis (Multi-AZ) (cache + sessions) | v S3 + CloudFront (static assets, documents) Why ECS Fargate over Kubernetes at this stage: Fargate removes all node management overhead. No node pools, no cluster upgrades, no kubelet debugging. For a 15-engineer team launching in three months, that overhead is not justified. Kubernetes becomes the right answer when the team has dedicated platform capacity and the complexity of their deployment patterns genuinely requires it — typically at 40+ engineers or with strong microservices requirements. Infrastructure as code from day one. Every resource in Terraform, state in S3 with DynamoDB locking, all changes through pull requests. ```hcl # State backend — set this up before writing a single resource terraform { backend "s3" { bucket = "company-terraform-state" key = "production/terraform.tfstate" region = "ap-south-1" dynamodb_table = "terraform-state-lock" encrypt = true } } ``` Monitoring from day one but deliberately minimal. CloudWatch for infrastructure metrics, a single Grafana dashboard for the four golden signals per service, PagerDuty for on-call. Do not build a full observability stack before you know which signals matter. Secrets in AWS Secrets Manager, accessed via IAM roles — not environment variables, not a config file. > 💡 **Green Flag:** The candidate explicitly discusses what they would NOT build and why. Knowing what to defer is a senior signal. Junior candidates try to show they know everything by including everything. > 🔴 **Red Flag:** Immediately designing Kubernetes with Istio and a full observability stack. The question asked for a three-month launch, not a year-two architecture. What you say at the end: "This is the three-month architecture. The twelve-month architecture looks meaningfully different — likely moving to EKS as the team grows and deployment complexity increases, adding distributed tracing once the service topology is established, and formalising the on-call rotation with SLOs as the user base grows. The principle is: build for where you are, design for where you are going." ### Designing for Multi-Region Availability ### The Question Your company wants to expand from one AWS region to three regions globally. Walk me through the architecture decisions you need to make and what changes. ### What the interviewer is testing Multi-region is a genuinely hard problem. They want to see that you understand it involves database consistency tradeoffs, not just replicating your current architecture three times. ### The core challenge Everything about multi-region is either a data problem or a latency problem. Compute is easy to distribute — you run the same containers in multiple regions. Data is where it gets hard. The question you must answer first: what is the consistency model? Option 1: Active-Passive One region handles all writes. Other regions read from replica. User writes go to primary region always. Read-heavy traffic can be served locally. Failover to secondary region if primary fails (minutes of downtime). Option 2: Active-Active All regions accept writes. Requires conflict resolution strategy. Much more complex — eventual consistency for some data. Sub-10ms latency for all users globally. For most companies below hyperscaler scale, active-passive is the correct answer. The consistency complexity of active-active is only justified when latency is a hard business requirement, which is rare. Active-passive architecture: ap-south-1 (Primary - Mumbai) - All write traffic - RDS Aurora (primary cluster) - Full application stack eu-west-1 (Secondary - Ireland) - Read traffic from European users - RDS Aurora read replica (async replication) - Application stack (read paths only) us-east-1 (Tertiary - Virginia) - Read traffic from US users - RDS Aurora read replica - Application stack (read paths only) Global routing with Route53: ```hcl # Latency-based routing — sends users to the nearest healthy region resource "aws_route53_record" "api" { zone_id = aws_route53_zone.main.zone_id name = "api.company.com" type = "A" latency_routing_policy { region = "ap-south-1" } set_identifier = "primary" alias { name = aws_lb.primary.dns_name zone_id = aws_lb.primary.zone_id evaluate_target_health = true } } ``` > 📌 **Remember:** Data residency requirements can override the technical architecture entirely. If you are storing Indian user data, RBI data localisation guidelines may require that data to stay in India. GDPR requires European user data to stay in the EU. Check compliance requirements before designing the data tier. The questions interviewers ask as follow-ups: What happens to in-flight requests when you fail over from primary to secondary? Answer: sessions stored in Redis need to be replicated. Idempotent request design means failed requests can be safely retried. How do you keep the secondary region warm? Answer: deploy the same application stack, run synthetic traffic against it, run load tests periodically. A region you have never tested under load will not perform correctly when you need it. How do you handle the cost? Answer: active-passive with smaller compute in secondary regions is significantly cheaper than active-active. Secondary region runs at reduced capacity and scales up during failover. ### The Build vs Buy Decision at Scale ### The Question Your team is evaluating whether to build an internal developer platform or buy one (like Backstage, Port, or Cortex). You have 200 engineers and 80 microservices. What is your recommendation and how do you make this decision? ### What the interviewer is testing Build vs buy is a recurring decision at senior level. They want to see a framework, not an opinion. ### The framework The right question is not "which tool is better" but "what does our team actually need, and what is the total cost of each option over three years?" Start by defining what problem you are solving. An internal developer platform (IDP) is the right answer when: * Engineers are losing significant time to infrastructure self-service tasks (creating new services, spinning up environments, checking service health) * Onboarding new engineers to the platform takes more than a week * There is no single place to see service ownership, runbooks, and deployment status If those problems do not exist at your scale, an IDP is premature. Evaluation criteria for build vs buy: | Dimension | Build (Backstage) | Buy (Port, Cortex) | |:----------|:------------------|:-------------------| | Initial cost | Low licence, high engineering | High licence, low engineering | | 3-year total cost | High — ongoing maintenance | Predictable SaaS pricing | | Customisation | Unlimited | Limited to vendor features | | Time to value | 3-6 months | 2-4 weeks | | Team dependency | Internal platform team required | Vendor-dependent | | Integration depth | Build exactly what you need | Limited to provided integrations | The honest answer for most companies: buy first, build only when you have exhausted what the vendor can provide. Backstage specifically: it is open-source but not free. The engineering cost to run a Backstage instance well — keeping plugins updated, building custom integrations, maintaining the plugin ecosystem — is typically one dedicated engineer minimum. At 200 engineers that may be justified. At 50 engineers it is not. > 💡 **Green Flag:** The candidate asks "what problem are we actually trying to solve" before recommending a solution. Many senior candidates skip this and go straight to tool comparison. What to say: "My recommendation is to buy a SaaS platform like Port or Cortex for the first year. At 200 engineers and 80 services, the time-to-value argument is strong — your platform team's time is better spent on reliability and developer experience improvements than maintaining plugin infrastructure. Revisit the build decision at 500 engineers when the customisation limits of the vendor become real constraints." ### CI/CD and Deployment Strategy ### Designing Zero-Downtime Database Migrations at Scale ### The Question You have a PostgreSQL database with 500 million rows in a critical table. You need to add an index on a column that has no index. The table is hit by 50,000 reads per second during peak. How do you do this with zero downtime? ### What the interviewer is testing Database operations at scale are where many senior candidates reveal gaps. Adding an index on a small table is trivial. On a 500 million row table under high load, a naive approach takes the database down. ### Why this is hard PostgreSQL's default CREATE INDEX acquires a ShareLock on the table. During the index build (which could take 30-60 minutes on a 500 million row table), all writes to the table are blocked. Your application times out. This is a production incident. ### The solution PostgreSQL provides CREATE INDEX CONCURRENTLY which builds the index without blocking reads or writes. ```sql -- DO NOT DO THIS on a large table under production load CREATE INDEX idx_orders_user_id ON orders(user_id); -- This blocks all writes for potentially 30-60 minutes -- DO THIS INSTEAD CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders(user_id); -- Builds without blocking writes -- Takes longer (roughly 2x) but safe under production load ``` > 📌 **Remember:** CONCURRENTLY has limitations. It cannot run inside a transaction block. It is more vulnerable to failure — if the build fails partway through, it leaves an INVALID index that must be manually dropped. Always check pg_indexes after completion. ```sql -- Verify the index built successfully SELECT schemaname, tablename, indexname, indexdef FROM pg_indexes WHERE tablename = 'orders' AND indexname = 'idx_orders_user_id'; -- Check for invalid indexes SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'orders' AND indexname = 'idx_orders_user_id' AND pg_index.indisvalid = false FROM pg_index JOIN pg_class ON pg_class.oid = pg_index.indexrelid WHERE pg_class.relname = 'idx_orders_user_id'; ``` Additional considerations for this scale: Run during low-traffic period. Even CONCURRENTLY adds I/O load. At 50,000 reads per second, running during peak adds latency. Schedule for 2-4 AM. Use connection pooling awareness. PgBouncer in transaction mode does not maintain session-level locks. PgBouncer in session mode does. Know your pooler behaviour before running long DDL. Monitor replication lag. If you have read replicas, the concurrent index build on primary generates significant WAL. Monitor replication lag on replicas during the operation — if they fall too far behind, you may need to pause and resume. ```bash # Monitor index build progress (PostgreSQL 12+) SELECT phase, blocks_done, blocks_total, round(100.0 * blocks_done / blocks_total, 1) AS percent_complete FROM pg_stat_progress_create_index WHERE relid = 'orders'::regclass; ``` What interviewers ask next: "What do you do if the concurrent index build fails?" Answer: drop the invalid index explicitly (`DROP INDEX CONCURRENTLY idx_orders_user_id`), investigate why it failed (disk space, replication issues, connection interruption), resolve the root cause, and rebuild. Never leave an INVALID index — the query planner may still try to use it. ### Cost Optimisation at Platform Scale ### The Question Your AWS bill is $400,000 per month. Your CTO asks you to reduce it by 30% without reducing reliability. Walk me through your approach. ### What the interviewer is testing FinOps is a senior competency that pure infrastructure engineers often lack. They want to see a systematic methodology, not "turn off unused resources." ### The approach A $120,000 monthly saving (30% of $400k) is a significant engineering project, not a quick win. Treat it like one — with a discovery phase, a prioritised backlog, and staged implementation. Phase 1: Measure (week 1-2) You cannot optimise what you cannot see. Use AWS Cost Explorer with cost allocation tags to break down spend by service, team, and environment. ```bash # Enable cost allocation tags if not already done # Every resource should be tagged with: team, environment, service # Example Terraform tagging strategy locals { common_tags = { team = var.team_name environment = var.environment service = var.service_name managed_by = "terraform" } } ``` Typical breakdown for a $400k/month bill: * EC2 / compute: 40-50% ($160-200k) * RDS / databases: 20-25% ($80-100k) * Data transfer: 10-15% ($40-60k) * S3 storage: 5-10% ($20-40k) * Everything else: 10-15% Phase 2: Quick wins (week 2-4, target 10-15% reduction) Rightsizing EC2 and RDS. AWS Compute Optimizer shows you instances that are consistently under-utilised. Downsizing a db.r5.4xlarge to db.r5.2xlarge on a database running at 20% CPU is a 50% cost reduction on that instance. ```bash # Get rightsizing recommendations aws compute-optimizer get-ec2-instance-recommendations \ --filters name=Finding,values=Overprovisioned ``` Turn off non-production environments outside business hours. Development and staging environments running 24/7 are pure waste. Schedule them to stop at 8 PM and restart at 8 AM — that is a 65% reduction in compute cost for those environments. ```python # Lambda function to stop/start EC2 by tag schedule # Run on EventBridge schedule import boto3 def lambda_handler(event, context): ec2 = boto3.client('ec2', region_name='ap-south-1') action = event['action'] ## 'start' or 'stop' ## Find instances tagged for scheduled shutdown instances = ec2.describe_instances( Filters=[ {'Name': 'tag:AutoStop', 'Values': ['true']}, {'Name': 'instance-state-name', 'Values': ['running'] if action == 'stop' else ['stopped']} ] ) instance_ids = [ i['InstanceId'] for r in instances['Reservations'] for i in r['Instances'] ] if instance_ids: if action == 'stop': ec2.stop_instances(InstanceIds=instance_ids) else: ec2.start_instances(InstanceIds=instance_ids) ``` Delete unattached EBS volumes and unused snapshots. These accumulate invisibly. A script to find and report them monthly, with a review process to delete or keep. Phase 3: Committed use and spot (month 2-3, target 10-15% additional reduction) Savings Plans for baseline compute. If you have a predictable baseline of EC2 usage (which any established production system does), Compute Savings Plans give 40-66% discount versus on-demand in exchange for a 1 or 3-year commitment to spend a certain dollar amount per hour. Spot instances for fault-tolerant workloads. CI/CD build workers, batch processing jobs, non-critical background workers — these are good spot candidates. Spot is 70-90% cheaper than on-demand. The risk is interruption (2-minute notice). Workloads must be interruptible. ```yaml # EKS node group with spot instances for CI workers apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig managedNodeGroups: - name: ci-workers-spot instanceTypes: ["m5.xlarge", "m5a.xlarge", "m4.xlarge"] spot: true minSize: 0 maxSize: 50 labels: workload-type: ci taints: - key: workload-type value: ci effect: NoSchedule ``` Phase 4: Architecture changes (month 3-6, target remaining savings) Data transfer costs. Outbound data transfer from AWS is expensive. If you are calling S3 frequently from EC2 in the same region, use VPC Endpoints for S3 (free) instead of going through the internet gateway (charged). If you are transferring data between regions, consider whether that traffic is necessary. S3 storage tiering. Data accessed infrequently should not live in S3 Standard. Use S3 Intelligent-Tiering for objects you are unsure about, or explicitly move data to S3-IA (Infrequent Access) or S3 Glacier for archive data. > 📌 **Remember:** Communicate cost reduction progress monthly to leadership. $120,000 in savings is significant engineering value. Make sure it is visible. ### Observability and SRE ### Designing a Multi-Burn-Rate Alert Strategy ### The Question Explain multi-window multi-burn-rate alerting and implement it for a payments API with a 99.9% availability SLO. ### What the interviewer is testing This is a filter question at senior level. Most engineers know what an SLO is. Few have actually implemented error budget alerting correctly. Multi-burn-rate alerting is the Google SRE approach that prevents both false positives and missed incidents. ### Why single-threshold alerting fails A naive SLO alert looks like: ```yaml # Bad: alert when error rate exceeds 0.1% (the SLO threshold) alert: PaymentsAPIErrorBudgetAtRisk expr: rate(http_errors_total{job="payments"}[5m]) / rate(http_requests_total{job="payments"}[5m]) > 0.001 ``` This has two problems. A brief spike lasting 2 minutes fires the alert even though it barely affects the monthly error budget. But a slow burn at 0.08% error rate — not enough to trigger the alert — can consume 80% of the monthly budget over 3 weeks without any alert firing. ### Burn rate explained If your SLO is 99.9%, your error budget is 0.1% of requests per month. A burn rate of 1 means you are consuming the error budget exactly as expected. A burn rate of 10 means you are burning through the monthly budget 10x faster — you will exhaust it in 3 days instead of 30. Multi-burn-rate alerting fires on burn rate, not raw error rate. And it uses two windows — a short window to detect fast burns early, and a long window to confirm the burn is sustained. ```yaml # Prometheus alerting rules for 99.9% SLO # Monthly error budget = 0.1% = 43.8 minutes of downtime groups: - name: payments_slo_alerts rules: ## Page immediately: burning budget 14x faster than expected ## Short window catches it fast, long window confirms it is real - alert: PaymentsCritical expr: | ( rate(http_requests_total{job="payments",status=~"5.."}[1h]) / rate(http_requests_total{job="payments"}[1h]) ) > (14 * 0.001) and ( rate(http_requests_total{job="payments",status=~"5.."}[5m]) / rate(http_requests_total{job="payments"}[5m]) ) > (14 * 0.001) for: 2m labels: severity: critical annotations: summary: "Payments API burning error budget 14x — exhausts in 2 hours" ## Page: burning 6x faster ## 5-hour window and 30-minute window - alert: PaymentsHighBurn expr: | ( rate(http_requests_total{job="payments",status=~"5.."}[6h]) / rate(http_requests_total{job="payments"}[6h]) ) > (6 * 0.001) and ( rate(http_requests_total{job="payments",status=~"5.."}[30m]) / rate(http_requests_total{job="payments"}[30m]) ) > (6 * 0.001) for: 15m labels: severity: warning annotations: summary: "Payments API burning error budget 6x — exhausts in 5 days" ## Ticket: slow burn that will exhaust budget in 3 weeks - alert: PaymentsSlowBurn expr: | ( rate(http_requests_total{job="payments",status=~"5.."}[1d]) / rate(http_requests_total{job="payments"}[1d]) ) > (3 * 0.001) for: 1h labels: severity: info annotations: summary: "Payments API slow burn — investigate before weekly review" ``` > 💡 **Green Flag:** The candidate explains what each alert is for and how they interact. Critical alerts page. High-burn creates a ticket. Slow-burn is a weekly review item. This shows they have thought about the operational workflow, not just the math. The error budget policy that makes this valuable: define in advance what the team does when different alerts fire. "If PaymentsCritical fires, we freeze all production deployments and treat it as a P1 incident. If PaymentsHighBurn fires, the on-call engineer investigates within 30 minutes. If PaymentsSlowBurn fires, it goes into the engineering review meeting." Without the policy, the alerts are just numbers. ### Running a Chaos Engineering Programme ### The Question Your CTO wants to start a chaos engineering practice. You have 40 microservices on Kubernetes. How do you start and what do you actually run? ### What the interviewer is testing Chaos engineering is a maturity signal. They want to see that you understand it requires a strong observability foundation first, and that you run it with clear hypotheses — not just random failure injection. ### The prerequisite question Before running chaos experiments, you need to be able to detect when something is wrong. If your observability is incomplete, chaos engineering just causes outages you cannot explain. The rule: if you cannot see the blast radius of a failure, you are not ready to inject it. Checklist before starting: * All services have RED dashboards (rate, errors, duration) * You have defined SLOs for at least your top 5 services * You have runbooks for the 10 most common failure scenarios * Your on-call team has practised incident response ### The progression Start with game days, not automated chaos. A game day is a scheduled, controlled exercise where you manually inject failure and observe what happens. Week 1 game day: Pod failure ```bash # Using Chaos Mesh (runs on Kubernetes) # Inject pod failure on a single non-critical service kubectl apply -f - <<EOF apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: pod-failure-test namespace: staging spec: action: pod-failure mode: one duration: "5m" selector: namespaces: - staging labelSelectors: app: notification-service EOF ``` Before running: write down your hypothesis. "When one notification-service pod fails, the other two pods should handle the traffic with less than 5% increase in error rate, and the failure should be transparent to users." After running: did reality match the hypothesis? If yes, your hypothesis about resilience was correct. If no, you found a real gap — maybe all three pods were on the same node (node failure would take all three), or maybe the service did not have proper retry logic. Month 2 game day: Network partition simulation ```yaml # Simulate network latency between order-service and payment-service apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: network-delay-test spec: action: delay mode: all selector: namespaces: - staging labelSelectors: app: order-service delay: latency: "500ms" correlation: "25" jitter: "100ms" target: selector: namespaces: - staging labelSelectors: app: payment-service mode: all duration: "10m" ``` Hypothesis: "When payment service calls have 500ms of added latency, order service should time out gracefully and return a user-friendly error rather than hanging indefinitely. The checkout flow should fail fast with a clear error, not cause a cascading timeout chain." > 🔴 **Red Flag:** Running chaos experiments in production before you have run them in staging. Running without a clear hypothesis is also a red flag — "let's see what breaks" is not chaos engineering, it is recklessness. Month 3 onwards: move to automated chaos for things you have already tested manually. Run pod failure injection on a schedule in staging. Build confidence in the system's resilience claims before moving to production. ### Security at Scale ### Designing a Zero-Trust Internal Network ### The Question Your company currently uses VPN for all internal access and network-level trust between services. Engineering leadership wants to move to zero-trust. What does that mean and how do you implement it? ### What the interviewer is testing Zero-trust is widely discussed and poorly understood. They want to see you can translate the concept into concrete implementation steps, not just recite the definition. ### What zero-trust actually means in practice Traditional perimeter security assumes: inside the network = trusted, outside = untrusted. Once an attacker is inside (via compromised credentials, lateral movement, supply chain attack), they can reach anything. Zero-trust replaces network location with identity as the trust boundary. Every request — whether from a user, a service, or an internal system — must prove its identity and be authorised for the specific action it is requesting. The network location is irrelevant. Implementation has three layers: Layer 1: Human access — replace VPN with identity-aware proxy ``` Before: User → VPN → access entire internal network After: User → Identity-Aware Proxy → specific service, specific action ``` Tools: Google BeyondCorp Enterprise, Cloudflare Access, or AWS Verified Access. Every request to an internal service requires a valid identity token (from your IdP — Okta, Azure AD, Google Workspace). The proxy validates the token and the user's device posture (is the device managed? Is it patched?) before forwarding the request. The result: a compromised password alone is not enough to access internal systems. The attacker also needs a managed device with valid posture. Layer 2: Service-to-service — mTLS with short-lived certificates ``` Before: Service A can call any Service B on the internal network After: Service A must present a valid certificate to call Service B, and Service B explicitly allows Service A in its policy ``` Implement with a service mesh (Istio or Linkerd) or with SPIFFE/SPIRE for certificate issuance without a full mesh. ```yaml # Istio AuthorizationPolicy — explicit allow list # By default, all traffic is denied between services # This policy allows payment-service to call the checkout endpoint only apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: payments-access-policy namespace: production spec: selector: matchLabels: app: payment-service rules: - from: - source: principals: ["cluster.local/ns/production/sa/order-service"] to: - operation: methods: ["POST"] paths: ["/v1/payments/process"] ``` Layer 3: Data access — attribute-based access control Which services can access which databases, and which specific tables or operations. Rather than "order-service can connect to the database," the policy is "order-service can SELECT from orders where orders.tenant_id = order-service's tenant claim." This is harder to implement but prevents the scenario where a compromised service can read all data in a shared database. Migration path from VPN to zero-trust: This is a 6-12 month project, not a weekend migration. Month 1-2: Deploy identity-aware proxy alongside VPN. Route non-critical internal tools (wikis, dashboards) through the proxy. Leave production access on VPN. Month 3-4: Enable mTLS in permissive mode (logs violations but does not block). Identify all services that use plain HTTP internally. Month 5-6: Switch mTLS to strict mode for non-production. Fix any services that fail. Month 7-9: Gradually move production services to strict mTLS. Enable AuthorizationPolicies with explicit allow-lists. Month 10-12: Decommission VPN access for internal services. Maintain VPN only for legacy systems that cannot be migrated. ### Platform and Team Leadership ### Designing an On-Call Rotation and Incident Response Process ### The Question Your team has never had a formal on-call rotation. You have 15 engineers and three critical services. Design the on-call process from scratch. ### What the interviewer is testing Senior engineers are expected to own reliability culture, not just reliability technology. This tests whether you can design a sustainable human process alongside the technical one. ### The principles A well-designed on-call process has three properties: it is fair (burden is shared equitably), it is effective (incidents are resolved quickly), and it is sustainable (engineers do not burn out). Rotation structure for 15 engineers, three critical services: Do not put 15 engineers in one rotation for three services. Segment by service ownership. Service A team (5 engineers): primary rotation = each engineer is on-call 1 week in 5 (roughly 10 weeks per year). Service B team (5 engineers): same. Service C team (5 engineers): same. This is far more sustainable than a 15-person rotation with a one-week-in-fifteen frequency — the infrequency means engineers are rusty each time they are on-call. Tooling requirements: ``` PagerDuty (or Opsgenie) for: - On-call schedule management - Alert routing (which alerts go to which team) - Escalation policies (if primary does not acknowledge in 10 minutes, page secondary) - On-call reporting (how many pages per week, MTTA, MTTR) ``` Alert severity policy: P1 (Critical): Pages primary immediately. Wakes them up. User-facing impact confirmed. P2 (High): Pages during business hours. Creates ticket outside hours. No confirmed user impact but risk of escalation. P3 (Medium): Creates ticket. Reviewed at next working day standup. The rule for P1 alert quality: if an engineer gets paged and their first response is "this alert fires all the time and nothing is actually broken," that alert is P2 at most and should be recategorised immediately. Persistent false-positive P1 alerts destroy on-call culture faster than anything else. Runbook requirement: every P1 alert must have a runbook. The runbook must answer three questions: 1. What does this alert mean in plain English? 2. What are the first three commands to run to diagnose it? 3. Who do you escalate to if you cannot resolve it in 30 minutes? Runbooks do not need to be long. They need to be accurate and current. Post-mortem process: Every P1 incident gets a post-mortem document within 48 hours. Blameless. Written by the incident commander (the on-call engineer who led the response). The template: ``` Incident: [Title] Date: [Date] Duration: [Start time to resolution] Impact: [What was affected, how many users, revenue impact if known] Timeline: [Chronological list of events — what was observed when, what actions were taken] Root cause: [Single sentence. Not "the database was slow" — "a missing index on the orders table caused full table scans under the new query pattern introduced in deployment 2.3.1"] Contributing factors: [Things that made this worse or harder to detect] Action items: [Specific, assigned, time-bound tasks. Not "improve monitoring" — "Add P99 latency alert for orders table queries by [owner] by [date]"] ``` The post-mortem review meeting: 30 minutes, weekly, all on-call engineers. Review the action items from last week's post-mortems. Identify the highest-priority reliability improvement for next week. > 💡 **Green Flag:** The candidate mentions on-call compensation or acknowledgement. Engineers who are woken up at 3 AM need recognition — whether that is on-call pay, comp days, or explicit acknowledgement in performance reviews. Ignoring this is how on-call becomes a culture problem. ### Evaluating and Hiring Senior Engineers ### The Question You are asked to design a technical interview process for a Senior DevOps Engineer role. What does it look like and what are you evaluating? ### What the interviewer is testing Can you think about hiring rigorously? And can you design an interview that finds real signal without wasting candidates' time? ### The structure A good senior DevOps interview has four components, each testing something distinct: **Component 1: Technical screen (45 minutes, phone)** Goal: filter for baseline technical depth without wasting full-day panel time. Ask two questions. One operational (walk me through debugging a Kubernetes pod that is crash-looping) and one design (how would you architect monitoring for a 20-service system?). Not to find the perfect answer but to hear how they think — do they ask clarifying questions, do they mention tradeoffs, do they know when they do not know something? **Component 2: System design (60 minutes, onsite or video)** Give a realistic scenario from your actual infrastructure. "We are running 30 services on EKS. We want to implement progressive delivery — canary deployments with automatic rollback based on error rate. Design the system." Evaluate: do they ask about team size, current tooling, existing observability? Do they propose a solution proportionate to the problem? Do they acknowledge what could go wrong with their proposal? **Component 3: Debugging simulation (45 minutes)** Give them access to a terminal connected to a broken staging environment. Tell them "the checkout service is degraded, 8% of requests are failing. Diagnose it." The environment has a deliberate bug — maybe a misconfigured resource limit causing OOMKills, maybe a database connection pool exhaustion. Evaluate: what commands do they run first? Do they narrate their thinking? Do they look in the right places? Do they stay calm? **Component 4: Behavioural and leadership (45 minutes)** Two or three STAR-format questions about past incidents, decisions, and team conflicts. "Tell me about a time your change caused a production incident." "Tell me about a technical decision you made that turned out to be wrong." Evaluate: do they take ownership or blame others? Do they show specific learning? Do they communicate clearly under the pressure of recounting a failure? > 🔴 **Red Flag:** Any interview process that requires a take-home project of more than 2-3 hours. Senior candidates are employed and cannot spend a weekend on your homework. Long take-homes filter for time availability, not competence. What to look for that separates senior from mid-level candidates: Senior candidates ask questions before answering. Mid-level candidates dive in. Senior candidates say "it depends" and then explain what it depends on. Mid-level candidates give the answer they know. Senior candidates talk about the humans in the system — the team, the change management, the communication. Mid-level candidates describe the technical solution. ### More Technical Questions **Q1. Your monorepo CI pipeline takes 55 minutes. Engineering leadership says fix it. What do you do?** Measure before changing anything. Pull per-step timing from your CI provider — GitHub Actions shows this natively. Most teams assume tests are the bottleneck when it is often dependency installation or sequential Docker builds. For a monorepo the fix is path-based change detection. No reason to build and test the payments service when only notifications changed: ```yaml - uses: dorny/paths-filter@v2 id: changes with: filters: | payments: - 'services/payments/**' notifications: - 'services/notifications/**' - name: Build payments if: steps.changes.outputs.payments == 'true' run: cd services/payments && docker build . ``` Beyond that, in order of impact: aggressive dependency caching (restore node_modules when lock file unchanged), parallelise affected services as a matrix build, structure Dockerfiles so dependency layers cache separately from source layers, separate fast gates (lint, unit tests under 3 minutes) from slow gates (integration, E2E — only on PRs to main). A well-optimised pipeline for a 1-2 service change in a 30-service monorepo should complete in 6-8 minutes. --- **Q2. How does Kubernetes handle pod eviction under node memory pressure?** The kubelet on each node monitors memory. When available memory drops below the eviction threshold, kubelet starts evicting pods — not the scheduler, the local kubelet. Eviction order is determined by QoS class: BestEffort pods (no requests/limits set) are evicted first. Burstable pods (actual usage exceeds requests) are evicted next. Guaranteed pods (requests equal limits) are evicted last. ```yaml # kubelet eviction configuration evictionHard: memory.available: "100Mi" evictionSoft: memory.available: "200Mi" evictionSoftGracePeriod: memory.available: "1m30s" ``` This is why setting resource requests and limits on every production pod matters — it determines survivability during node pressure. A pod with no requests is treated as lowest priority and evicted first regardless of how important it is to your business. --- **Q3. What is the difference between HPA scaling on CPU versus scaling on custom metrics? When would you use each?** CPU-based HPA is simple and works well for compute-bound services — APIs where processing a request consumes CPU proportionally to load: ```yaml metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 ``` CPU scaling breaks down for I/O-bound services (waiting on database, network calls) where CPU stays low even under high load. It also breaks for queue-consuming workers — the right signal is queue depth, not CPU. Custom metrics via the Kubernetes metrics API solve this. KEDA (Kubernetes Event-Driven Autoscaling) is the cleanest implementation: ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject spec: scaleTargetRef: name: order-processor triggers: - type: rabbitmq metadata: queueName: order-events value: "30" ## scale up when queue has more than 30 messages per replica ``` Use CPU scaling for synchronous request-response services. Use custom metrics for queue consumers (scale on queue depth), services with external rate limits (scale on API quota remaining), or any service where CPU is not proportional to work. --- **Q4. Explain Terraform workspaces and when you should use them versus separate state files.** A Terraform workspace creates an isolated state file within the same backend configuration. Switching workspaces switches which state file is active: ```bash terraform workspace new staging terraform workspace select production terraform workspace list ``` Each workspace gets its own state file at a path like `env:/staging/terraform.tfstate`. When workspaces are appropriate: managing multiple nearly-identical environments (dev, staging, production) from the same Terraform codebase where the only difference is variable values like instance size or replica count. When separate state files are better: when environments have meaningfully different infrastructure (production has a multi-AZ RDS, staging has a single-AZ), when different teams manage different environments, or when you want strong isolation — a mistake in the staging workspace should not be possible to accidentally affect production state. The practical rule most senior engineers follow: workspaces for truly identical environments at different scales. Separate state files in separate directories for environments that differ structurally or are owned by different teams. --- **Q5. What is AWS Service Control Policy and how is it different from IAM?** IAM policies control what a specific user, role, or service can do. SCPs (Service Control Policies) in AWS Organizations set the maximum permissions boundary for entire AWS accounts — they apply before IAM policies are evaluated. An SCP is a guardrail, not an allow. Even if an IAM user has AdministratorAccess, an SCP can prevent them from performing specific actions: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Action": [ "ec2:*", "rds:*" ], "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["ap-south-1", "ap-southeast-1"] } } } ] } ``` This SCP denies all EC2 and RDS operations outside Mumbai and Singapore — regardless of what IAM policies say. Even a root user in the account cannot launch EC2 in us-east-1 while this SCP is active. Use SCPs for: data residency requirements (block all operations outside approved regions), cost control (prevent launching expensive instance types), security baseline (block disabling CloudTrail, block removing VPC flow logs). SCPs apply to the entire account and are managed centrally by the platform team — individual account admins cannot override them. --- **Q6. How do you implement zero-downtime schema migrations for a PostgreSQL table with 500 million rows?** Adding a column with a default value, adding an index, or changing a column type on a 500 million row table using standard DDL acquires locks that block writes for 30-60 minutes. That is a production outage. For indexes, use CONCURRENTLY: ```sql -- Blocks writes for potentially 60 minutes on 500M rows CREATE INDEX idx_orders_user_id ON orders(user_id); -- Builds without blocking writes -- takes 2x longer but safe CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders(user_id); ``` For column additions with defaults (PostgreSQL 11+), adding a NOT NULL column with a constant default is instant because PostgreSQL stores the default in metadata rather than rewriting every row: ```sql -- Safe in PostgreSQL 11+ -- metadata-only change ALTER TABLE orders ADD COLUMN status_v2 VARCHAR(50) DEFAULT 'pending' NOT NULL; ``` For column type changes or adding NOT NULL to existing nullable columns, use the expand-contract pattern across three deployments: Deploy 1: Add the new column as nullable. Application writes to both old and new column. Deploy 2: Backfill the new column for existing rows in batches (not one UPDATE on 500M rows — that locks the table): ```sql -- Backfill in batches of 10,000 to avoid long locks DO $$ DECLARE batch_size INT := 10000; last_id BIGINT := 0; max_id BIGINT; BEGIN SELECT MAX(id) INTO max_id FROM orders; WHILE last_id < max_id LOOP UPDATE orders SET new_column = derive_value(old_column) WHERE id > last_id AND id <= last_id + batch_size; last_id := last_id + batch_size; PERFORM pg_sleep(0.1); -- brief pause to avoid I/O saturation END LOOP; END $$; ``` Deploy 3: Application reads from new column. Drop old column. --- **Q7. What is eBPF and how is it changing Kubernetes networking and observability?** eBPF (extended Berkeley Packet Filter) is a kernel technology that runs sandboxed programs inside the Linux kernel without modifying kernel source or loading modules. It attaches to kernel hooks — network packets arriving, system calls being made — and your program runs in a safe VM inside the kernel. In networking, Cilium replaces kube-proxy entirely using eBPF. Instead of iptables rules (which degrade badly past 10,000 rules), Cilium implements Kubernetes Service routing directly in the kernel. The result is faster packet processing, lower CPU overhead, and NetworkPolicies with actual per-connection visibility. In observability, tools like Pixie use eBPF to automatically capture HTTP, gRPC, MySQL, and Kafka request/response data without any application instrumentation — zero code changes, kernel-level interception: ```bash # Pixie shows actual HTTP calls from payment-service # without touching any application code px run px/http_data_filtered -- -service payment-service -start_time -5m ``` The traditional observability approach requires either application-side instrumentation or a sidecar proxy (Envoy in a service mesh) adding 50-200MB per pod. eBPF achieves the same visibility at kernel level with no sidecar overhead. For senior interviews: eBPF is replacing sidecar-based service meshes for teams that want observability and network policy without the Istio overhead. Knowing this shows you track where the industry is heading. --- **Q8. Design a secrets rotation system for 50 microservices.** Manual rotation at 50 services means it will not happen consistently. The architecture is a centralised secrets manager with automatic rotation, and services that pull secrets dynamically rather than having them baked in at startup. For database credentials, HashiCorp Vault dynamic secrets are the highest-security approach. Vault generates a unique credential pair per service instance with a TTL — when it expires, the credential is automatically revoked: ```bash vault write database/roles/order-service \ db_name=production-db \ creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT ON orders TO \"{{name}}\";" \ default_ttl="1h" \ max_ttl="24h" ``` Each pod authenticates to Vault via its Kubernetes service account token, gets a credential that expires in 1 hour, and renews before expiry. A compromised credential is useless after 1 hour. For TLS certificates, cert-manager handles issuance and rotation automatically: ```yaml apiVersion: cert-manager.io/v1 kind: Certificate spec: secretName: service-tls duration: 24h renewBefore: 8h issuerRef: name: vault-issuer ``` The delivery pattern: never read secrets from environment variables set at container start. Use the Secrets Store CSI driver to mount secrets as files that update when rotated. The application reads from the file. The CSI driver updates the file on rotation. --- **Q9. How do you implement and manage RBAC for a 50-engineer Kubernetes cluster?** Bind to groups, not individual users. When you bind RBAC to individual user accounts, every hire and departure requires a manual RBAC change. Bind to groups from your identity provider (Okta, Azure AD) — add someone to the group, access is granted. Remove them, access is revoked instantly: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: developer-binding namespace: backend-production subjects: - kind: Group name: backend-engineers ## group in your IdP roleRef: kind: Role name: developer ``` Namespace isolation per team. Each team gets their own namespaces. Their Role only applies in those namespaces — they cannot affect other teams' workloads: ```yaml # Backend developers can manage pods and configmaps in their namespace # They have no access to frontend-production or data-production rules: - apiGroups: [""] resources: ["pods", "pods/log", "configmaps"] verbs: ["get", "list", "watch", "create", "update", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch", "update", "patch"] # No cluster-level resources. No exec into pods (separate explicit grant). ``` For CI/CD service accounts, minimal permissions only. The CI pipeline only needs to update deployment image tags: ```yaml # CI service account can only update the specific deployment rules: - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "patch", "update"] resourceNames: ["payment-service"] ## only this specific deployment ``` Audit quarterly using `kubectl auth can-i --list` to verify permissions are still appropriate. Remove anything that is no longer needed. --- **Q10. What is chaos engineering and how do you run it safely?** Chaos engineering is deliberately injecting failure into a system to verify it behaves as you expect under adverse conditions. The key word is deliberately — you run controlled experiments with defined hypotheses, not random destruction. The prerequisite before any chaos experiment: your observability must be good enough to see the blast radius. If you cannot detect the impact of the failure you are injecting, you are not running chaos engineering — you are causing outages you cannot explain. The progression: start with game days in staging, not automated chaos in production. A game day is a scheduled exercise where you manually inject failure and observe what happens. Write the hypothesis before running: "When one payment-service pod fails, the other two should absorb traffic with less than 5% error rate increase." ```yaml # Chaos Mesh on Kubernetes — kill one pod in staging apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: payment-pod-failure namespace: staging spec: action: pod-failure mode: one duration: "5m" selector: labelSelectors: app: payment-service ``` After the experiment: did reality match the hypothesis? If yes, you have confirmed your resilience claim. If no, you found a real gap — maybe all three replicas were on the same node, or retry logic was missing. Month 2: network partition simulation — inject latency between services. Month 3: node failure. Only move to production chaos experiments for things you have already validated in staging. The goal is confidence, not surprise. --- **Q11. Explain multi-burn-rate alerting and implement it for a 99.9% availability SLO.** Standard SLO alerting fires when error rate exceeds the SLO threshold. This creates two problems: a 2-minute spike fires the alert even though it barely affects the monthly budget, but a slow burn at slightly below the threshold can consume 80% of budget over 3 weeks without any alert. Burn rate fixes this. If your 99.9% SLO means 0.1% error budget per month, a burn rate of 1 means consuming budget at exactly the expected rate. A burn rate of 14 means consuming the monthly budget 14x faster — exhausted in 2 days. Multi-window means using two time windows per alert — a long window to detect the trend and a short window to confirm it is not a transient spike: ```yaml # Page immediately: 14x burn rate confirmed in both 1h and 5m windows - alert: PaymentsCritical expr: | (rate(http_errors_total{job="payments"}[1h]) / rate(http_requests_total{job="payments"}[1h])) > (14 * 0.001) and (rate(http_errors_total{job="payments"}[5m]) / rate(http_requests_total{job="payments"}[5m])) > (14 * 0.001) for: 2m labels: severity: critical # Page: 6x burn rate over 6h window confirmed in 30m window - alert: PaymentsHighBurn expr: | (rate(http_errors_total{job="payments"}[6h]) / rate(http_requests_total{job="payments"}[6h])) > (6 * 0.001) and (rate(http_errors_total{job="payments"}[30m]) / rate(http_requests_total{job="payments"}[30m])) > (6 * 0.001) for: 15m labels: severity: warning # Ticket: slow burn over 24h - alert: PaymentsSlowBurn expr: | (rate(http_errors_total{job="payments"}[1d]) / rate(http_requests_total{job="payments"}[1d])) > (3 * 0.001) for: 1h labels: severity: info ``` Critical alert pages. High-burn creates a ticket. Slow-burn goes into the weekly engineering review. Without defining what action each alert triggers, the alerts are just numbers. --- **Q12. How do you implement platform security hardening for a Kubernetes cluster?** Layer by layer. Each layer assumes the layer above it can be compromised. Node hardening: run a minimal immutable OS (Bottlerocket). No SSH by default — use SSM Session Manager. The node is not modified manually. Pod Security Standards: enforce restricted mode at namespace level — prevents root containers, host path mounts, privileged containers: ```bash kubectl label namespace production \ pod-security.kubernetes.io/enforce=restricted ``` Network policies: default deny all traffic, explicitly allow what is needed: ```yaml # Default deny in every namespace apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all spec: podSelector: {} policyTypes: [Ingress, Egress] ``` Image signing and scanning: every image in production must come from your private registry, be scanned (Trivy in CI blocks on CRITICAL CVEs), and be signed (Cosign). Kyverno enforces the policy: ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-signed-images spec: validationFailureAction: Enforce rules: - name: check-signature match: resources: kinds: ["Pod"] verifyImages: - image: "registry.company.com/*" key: |- -----BEGIN PUBLIC KEY----- MFkwEwYH... ``` Runtime threat detection: Falco watches system calls and alerts when a container does something unexpected — opening a shell, writing to /etc/passwd, making outbound network calls to unknown IPs: ```yaml - rule: Shell spawned in production container condition: container.id != host and proc.name in (bash, sh, zsh) and spawned_process and k8s.ns.name = "production" output: "Shell in production (user=%user.name pod=%k8s.pod.name image=%container.image.repository)" priority: WARNING ``` --- **Q13. What is the difference between a liveness probe, readiness probe, and startup probe?** Liveness: is this container still alive? Failure kills and restarts the container. Use for detecting deadlocks or corrupted internal state the app cannot recover from. Do not check external dependencies — if the database is down and liveness fails, you get a restart storm. ```yaml livenessProbe: httpGet: path: /health/live ## 200 if app is alive internally, regardless of deps port: 8080 initialDelaySeconds: 30 ## wait for startup periodSeconds: 10 failureThreshold: 3 ``` Readiness: is this container ready to serve traffic? Failure removes the pod from Service endpoints — traffic stops, pod is not restarted. Use for checking dependencies, cache warmup, connection establishment. A pod that is live but not ready absorbs no traffic until it recovers: ```yaml readinessProbe: httpGet: path: /health/ready ## 200 only when dependencies are up and cache is warm port: 8080 periodSeconds: 5 failureThreshold: 3 successThreshold: 2 ## require 2 consecutive passes to re-add to endpoints ``` Startup: has this container finished starting? Runs only at startup. Once it passes once, it is disabled and liveness/readiness take over. Solves slow-starting applications (JVM warmup, large data loads) where initialDelaySeconds is not predictable: ```yaml startupProbe: httpGet: path: /health/started port: 8080 failureThreshold: 30 ## allow up to 5 minutes (30 * 10s) for startup periodSeconds: 10 ``` > 🔴 **Common Mistake:** Liveness probe checks database connectivity. If the database goes down, all pods restart simultaneously, which makes the database recovery harder and your service unavailable for longer. --- **Q14. How would you design a self-service developer platform for 200 engineers?** Start by defining what problem you are solving. The right trigger for an internal developer platform is when engineers are losing significant time to infrastructure self-service tasks — creating new services, spinning up environments, understanding who owns what. If that friction does not exist, the platform is premature. For 200 engineers the core capabilities are: service catalogue (what services exist, who owns them, their health), environment provisioning (create a preview environment in under 5 minutes), and deployment visibility (what version is running where, who deployed it). Build vs buy: at 200 engineers, buy a SaaS product like Port or Cortex first. They deploy in days. The engineering cost of running Backstage — one dedicated engineer minimum — is not justified until you have exhausted what the vendor provides. The environment provisioning flow: ``` Developer runs: make env PR=feature/checkout-v2 | v CI creates Kubernetes namespace with resource quotas Deploys all services using branch image tags Creates DNS: checkout-v2.preview.company.com Posts Slack notification with URL | v PR merged → namespace deleted automatically 48h TTL → namespace deleted if PR is abandoned ``` The two things that make this actually useful: it runs in under 5 minutes (engineers will not use a 20-minute provisioning flow), and cleanup is automatic (without TTLs you end up with 200 stale environments consuming cluster capacity). --- **Q15. What is GitOps and when is it not the right answer?** GitOps is a deployment model where Git is the single source of truth for desired cluster state. A controller running inside the cluster (ArgoCD, Flux) watches the Git repository and reconciles the cluster to match. No external system holds cluster credentials. The security benefit: in push-based CI, your CI system has kubectl credentials. If the CI system is compromised, the attacker has cluster access. In GitOps, no external system touches the cluster — the cluster reaches out to Git. Compromising CI lets an attacker push a bad image to your registry but not execute commands against the cluster. ArgoCD also provides drift detection — if someone runs kubectl apply manually in production, ArgoCD detects the deviation from Git and can auto-correct it: ```bash argocd app diff payment-service ## show what drifted from Git argocd app sync payment-service ## reconcile back to Git state ``` When GitOps is not the right answer: for teams with fewer than 5-10 services and simple deployment patterns, GitOps adds overhead that is not justified. The image-tag-update-commit workflow is awkward for fast iteration. Push-based CI with scoped short-lived credentials (OIDC federation, not static keys) is simpler and perfectly adequate. Add GitOps when the audit trail, drift detection, and credential elimination genuinely matter — typically at larger scale or when compliance requires it. --- **Q16. How do you manage configuration across 50 microservices with different config per environment?** Separate non-sensitive config from secrets. Non-sensitive config lives in Git. Secrets live in a secrets manager. Never mix them. For non-sensitive config, Helm values files per environment: ``` services/payment-service/ Chart.yaml values.yaml # defaults values-staging.yaml # staging overrides values-production.yaml # production overrides ``` ```bash helm upgrade payment-service ./payment-service \ -f values.yaml \ -f values-production.yaml \ --namespace production ``` For secrets, the Secrets Store CSI driver mounts secrets from AWS Secrets Manager as files. The pod reads from the file. When the secret rotates, the file updates automatically without a pod restart: ```yaml volumes: - name: secrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: payment-service-secrets ``` For shared config used by many services (database endpoints, service discovery), a shared ConfigMap avoids duplicating the value 50 times. One update propagates to all services on next restart. The drift problem: staging and production configs diverge when someone hotfixes production and forgets to update Git. Prevent this by treating config changes as code changes — all modifications go through PRs with staging and production configs diffed as part of the release process. --- **Q17. Explain how you would architect a multi-region deployment for a payments API.** Multi-region architecture is fundamentally a data problem, not a compute problem. Compute is easy to distribute. Data consistency is where the tradeoffs live. First question: active-active or active-passive? Active-active means all regions accept writes — requires conflict resolution, eventual consistency for some operations, much more complex. Active-passive means one region handles writes, others serve reads — simpler, consistent, but writes always route to primary region. For payments: active-passive. Payments require strong consistency. A double-charge caused by a write conflict in active-active is a business catastrophe. ``` ap-south-1 (Primary — Mumbai): all writes, all reads | Continuous async replication (< 1 second lag) | v ap-southeast-1 (DR — Singapore): read-only replica, standby for failover ``` Route53 latency-based routing with health checks handles automatic failover. When the primary region health check fails, Route53 automatically routes to the DR region. For the 5-minute RPO: RDS Aurora Global Database provides replication lag typically under 1 second. The DR cluster can be promoted to primary in 1-2 minutes with negligible data loss. For the 1-hour RTO: pre-provision the DR infrastructure (EKS cluster, load balancers, networking). When failover happens, you scale up existing infrastructure rather than building from scratch. The 60-minute window breaks down as: 5 minutes detection and decision, 10 minutes database promotion, 15 minutes DNS propagation, 30 minutes scale-up and smoke tests. Test quarterly. Every DR drill finds something — a hardcoded primary region endpoint, an expired certificate in DR, a security group that was not replicated. --- **Q18. What is a Kubernetes operator and when would you build one?** An operator is a custom controller that encodes operational knowledge about a specific application into Kubernetes. It extends the API with custom resource types and manages their full lifecycle — not just at install time like Helm, but continuously, reacting to state changes. Helm is a package manager. It installs and upgrades. After installation, Helm is done. If the database primary fails, Helm does nothing. An operator runs continuously and reconciles. When the database primary fails, the operator detects it and promotes a secondary. When you scale the custom resource, the operator creates the new replicas correctly. When you upgrade the database version, the operator executes the correct upgrade sequence for that database engine. ```yaml # Developer declares what they want apiVersion: platform.company.com/v1 kind: InternalDatabase metadata: name: orders-db spec: engine: postgresql storage: "100Gi" replicas: 3 ``` The operator handles: provisioning the RDS instance, configuring replication, creating the Kubernetes Secret with connection details, scheduling backups, handling failover, and managing version upgrades. Build an operator when: you have an internal platform resource that multiple teams need with complex lifecycle (database provisioning, message queue management, certificate issuance), or you need continuous reconciliation that reacts to cluster events — not just install-time setup. Do not build an operator when: Helm plus a few CronJobs would work, when your team lacks Go/Kubernetes expertise to maintain it, or when the lifecycle complexity is actually simple. --- **Q19. What is your approach to FinOps? How do you reduce cloud spend without reducing reliability?** FinOps is the practice of making cloud cost a shared engineering responsibility rather than a surprise at month end. As a senior engineer you own the visibility, the tooling, and the recommendations — not the budget decisions, but the data that informs them. The foundation is cost attribution. Every resource tagged with team, environment, service. Cost Explorer broken down by these tags. Without attribution you cannot have the conversation about which team's service caused the bill to jump. Quick wins (first month, target 10-15% reduction): Rightsizing — AWS Compute Optimizer identifies instances running at 20% CPU. A db.r5.4xlarge at 20% CPU is a straightforward downsize. Stop non-production environments outside business hours — dev and staging running 24/7 is pure waste, schedule stop at 8 PM and start at 8 AM for a 65% compute reduction on those environments. Medium-term (month 2-3, target additional 10-15%): Committed use — Savings Plans for any compute that has run consistently for 3+ months. 40-66% discount for a 1-year hourly spend commitment. Spot instances for fault-tolerant workloads — CI build workers, batch jobs. 70-90% cheaper, requires handling 2-minute interruption notice. Architecture changes (month 3-6): Data transfer is the hidden cost. Calls from EC2 to S3 in the same region should use VPC Endpoints (free). S3 Intelligent-Tiering for data with unknown access patterns. Logs and archives to Glacier after 90 days. Report progress monthly with before/after numbers. Cost reduction is engineering business value that needs to be visible to leadership. --- **Q20. How do you design and run a meaningful DR drill?** A DR drill that does not actually fail over is not a drill — it is a documentation review. The drill has to involve actually promoting the DR database and routing traffic to the secondary region to be meaningful. Pre-drill preparation: notify stakeholders of the maintenance window, confirm the runbook is current, verify DR infrastructure is healthy, ensure the team knows their roles (one person executes, one person monitors metrics, one person communicates status). The drill execution sequence: ```bash # 1. Confirm primary is healthy (baseline) curl https://api.company.com/health # 2. Simulate primary region failure (stop accepting new connections) # In a drill, you can do this by updating Route53 health check to fail # rather than actually taking down the region # 3. Verify Route53 automatic failover fires # Watch DNS propagation in real time watch -n5 "dig api.company.com +short" # 4. Promote DR database (the most critical step) aws rds failover-global-cluster \ --global-cluster-identifier payments-global-cluster \ --target-db-cluster-identifier payments-dr-cluster # 5. Smoke test the DR region curl https://api-dr.company.com/health curl -X POST https://api.company.com/test-payment ... # 6. Measure actual RTO — time from simulated failure to smoke tests passing ``` What every drill finds: something. Common discoveries include DNS TTL that was set too high (5 minutes, should be 30 seconds for faster failover), a hardcoded primary region database endpoint in application config, an SSL certificate that expired in the DR environment, a security group that allowed traffic from the primary VPC CIDR only (not the DR VPC), or a service account permission that existed in primary but not DR. These are cheap to find in a drill. Expensive to find during an actual disaster at 2 AM. Document every drill result and the action items. Run quarterly. Rotate who executes the drill so multiple engineers know the procedure. --- **Q21. What is SBOM and why is it becoming mandatory?** SBOM (Software Bill of Materials) is a formal, machine-readable inventory of every component in your software — every open-source library, every dependency, every transitive dependency, and the version of each. Why it is becoming mandatory: the Log4Shell vulnerability in 2021 affected tens of thousands of organisations. Many of them did not know they were using Log4j because it was a transitive dependency — their application did not use it directly, but a library they used included it. An SBOM would have made the answer to "are we affected?" a query rather than a manual audit. US Executive Order 14028 (2021) requires SBOM for software sold to the US federal government. EU Cyber Resilience Act extends similar requirements to the European market. Financial services regulators are starting to ask for SBOM as part of vendor risk assessments. Generating SBOMs in the CI pipeline: ```bash # Syft generates SBOM from a container image syft registry.company.com/payment-service:1.2.3 -o spdx-json > sbom.json # Grype checks the SBOM against known CVEs grype sbom:./sbom.json --fail-on high # Pipeline fails if any HIGH or CRITICAL CVEs found in the SBOM ``` The SBOM should be: generated for every release, stored alongside the image in the registry, versioned so you can query "which versions of our software contain log4j 2.14.1", and checked on release AND continuously (new CVEs are discovered for old library versions). The practical value for your team today: when the next Log4Shell-class vulnerability hits, you answer "are we affected?" in 5 minutes by querying your SBOM store, rather than 5 days of manual dependency auditing. --- **Q22. How does Kubernetes rolling deployment work at the pod level — what exactly happens step by step?** When you run kubectl rollout or update a Deployment image, Kubernetes executes the rolling update through the ReplicaSet controller. The exact behaviour is controlled by maxSurge and maxUnavailable. ```yaml strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 ## can have 1 extra pod above desired count temporarily maxUnavailable: 0 ## never drop below desired replica count ``` For a Deployment with 5 replicas, maxSurge: 1, maxUnavailable: 0: Step 1: Controller creates 1 new pod with the new version. Total: 5 old + 1 new = 6 pods. Step 2: New pod starts. kubelet pulls the image, starts the container, runs the startup probe (if configured). Step 3: Readiness probe passes on the new pod. Kubernetes adds it to the Service endpoints. Now 6 pods are serving traffic. Step 4: Controller terminates 1 old pod. Sends SIGTERM, waits for preStop hook (if configured), waits for terminationGracePeriodSeconds, then sends SIGKILL if still running. Total: 4 old + 1 new = 5 pods. Step 5: Repeat until all 5 pods are the new version. At no point does the available count drop below 5 (maxUnavailable: 0). The rollout pauses if a new pod never passes its readiness probe — Kubernetes will not terminate old pods to maintain availability. This is why readiness probes are critical for zero-downtime deployments. Without one, Kubernetes considers a pod ready the moment the container starts — before your application has established database connections, loaded caches, or finished initialising. Traffic arrives at a pod that is not ready to serve it. ```yaml readinessProbe: httpGet: path: /health/ready ## return 200 only when genuinely ready port: 8080 initialDelaySeconds: 10 periodSeconds: 5 failureThreshold: 3 successThreshold: 1 ``` --- **Q23. What is the difference between Prometheus recording rules and alerting rules?** A recording rule precomputes an expensive PromQL expression and stores the result as a new metric. When you query the precomputed metric later, Prometheus returns the stored result instead of computing it from raw data — fast regardless of the time range. Recording rules are critical for: dashboard queries that aggregate over long time ranges (querying 30 days of data at query time is slow), frequently used subexpressions that appear in multiple alerting rules, and cardinality reduction (aggregate high-cardinality raw metrics into lower-cardinality summaries). ```yaml # recording rule — precompute error rate ratio groups: - name: payments.rules rules: - record: job:payments_error_rate:ratio_rate5m expr: | sum(rate(http_requests_total{job="payments",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="payments"}[5m])) ``` Now `job:payments_error_rate:ratio_rate5m` is a stored metric that Prometheus updates every evaluation interval. Dashboard queries and alerting rules reference this metric instead of computing the ratio from scratch every time. An alerting rule evaluates a PromQL expression on a schedule and fires an alert when the expression returns results: ```yaml - alert: PaymentsErrorRateHigh expr: job:payments_error_rate:ratio_rate5m > 0.01 for: 5m labels: severity: critical annotations: summary: "Payments error rate {{ $value | humanizePercentage }}" ``` The for: 5m clause means the condition must be true for 5 continuous minutes before the alert fires — prevents alerts from firing on transient spikes. Recording rules should be used aggressively for any expression that appears in multiple places or that queries long time ranges. A Grafana dashboard that computes a 7-day error rate at query time will time out on a large Prometheus. The same query against a recording rule returns instantly. --- **Q24. How do you handle a full disk on a Kubernetes worker node?** A node with a full disk will reach DiskPressure condition. kubelet starts evicting pods to reclaim space, and the node becomes unavailable for new scheduling. Immediate response: ```bash # Confirm disk pressure kubectl describe node <node-name> | grep -A5 "Conditions:" # Look for: DiskPressure True # SSH to node (via SSM) aws ssm start-session --target <instance-id> # Find what is consuming space df -h ## which filesystem is full du -sh /* 2>/dev/null | sort -rh | head 20 ``` The most common causes on Kubernetes nodes: Container logs: each pod writes to /var/log/containers on the node. A chatty pod with no log rotation can fill this quickly. Check which pod is the culprit: ```bash du -sh /var/log/containers/* | sort -rh | head 10 ``` Docker/containerd image cache: pulling many images over time fills /var/lib/containerd. Remove unused images: ```bash crictl rmi --prune ## remove all unused images # or via containerd ctr -n k8s.io images rm $(ctr -n k8s.io images ls -q | grep -v "sha256:") ``` Application data written to the node filesystem: a misconfigured application writing to the container's writable layer or to a host path mount will fill disk. Identify with: ```bash du -sh /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/*/fs | sort -rh | head 20 ``` Immediate remediation: delete the largest unnecessary files, then cordon and drain the node to move workloads elsewhere and rebuild from scratch with a larger volume. Prevention: configure log rotation in your container runtime, set storage limits on node-critical paths, monitor disk usage with an alert at 80% before it becomes critical. --- **Q25. What is a Kubernetes admission webhook and what are the security implications of running one?** An admission webhook is an HTTPS endpoint that receives API objects before they are persisted to etcd. Mutating webhooks can modify the object. Validating webhooks can approve or reject it. Both run for every matching API request. Common use cases: injecting the Istio sidecar into every pod (mutating), enforcing resource limits on every container (validating), requiring image signatures (validating), blocking deployment to production namespaces without an approved PR label (validating). ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: resource-limits-enforcer webhooks: - name: limits.company.com clientConfig: service: name: admission-webhook namespace: platform-tools path: "/validate-pod" rules: - operations: ["CREATE", "UPDATE"] resources: ["pods"] failurePolicy: Fail ## if webhook is unreachable, reject all pod creates sideEffects: None admissionReviewVersions: ["v1"] ``` The security implications: The webhook server has access to every API object in every matching request — including Secrets if you configure it to watch Secrets. A compromised webhook server is a significant security risk. Run it in a dedicated namespace with minimal RBAC. The failurePolicy setting is critical. `Fail` means if the webhook server is down, all pod creates are rejected — potentially blocking recovery from an outage. `Ignore` means if the webhook is down, the request passes without validation. Neither is perfect. Most production deployments use `Fail` for security-critical webhooks (image signature enforcement) and `Ignore` for convenience webhooks (adding labels). The webhook must be highly available. A validating webhook that enforces security policy must not be a single point of failure. Run 3+ replicas with a PodDisruptionBudget. > ⚠️ **Security:** The webhook server must use TLS — the API server will not send requests to HTTP. Use cert-manager to manage the webhook's certificate and handle rotation. --- **Q26. Explain the concept of back pressure in distributed systems and how you handle it.** Back pressure is the mechanism by which a downstream service signals to an upstream service that it is overwhelmed — forcing the upstream to slow down rather than continue flooding the downstream with requests it cannot handle. Without back pressure: Service A sends 10,000 requests per second to Service B. Service B can only handle 1,000. Service B's queue fills up, memory exhausts, the process crashes. Service A either retries (making it worse) or drops responses. Cascading failure. With back pressure: Service B signals "I am at capacity" and Service A throttles its sending rate to match Service B's processing capacity. HTTP-based back pressure: Service B returns 429 (Too Many Requests) with a Retry-After header. Service A implements exponential backoff and respects the retry window: ```python def call_service_b(payload, max_retries=5): for attempt in range(max_retries): response = requests.post(service_b_url, json=payload) if response.status_code == 429: retry_after = int(response.headers.get('Retry-After', 5)) time.sleep(retry_after * (2 ** attempt)) ## exponential backoff continue return response raise Exception("Service B unavailable after retries") ``` Queue-based back pressure: use a message queue between services. Service A writes to the queue at whatever rate it produces. Service B consumes from the queue at the rate it can process. The queue absorbs the burst. If the queue depth grows beyond a threshold, trigger scaling of Service B or alert that Service A is producing faster than Service B can consume. Circuit breaker as back pressure: when a downstream service is consistently failing or slow, a circuit breaker opens and stops sending requests — giving the downstream time to recover. Resilience4j for JVM services, Envoy at the service mesh level: ```yaml # Istio circuit breaker apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule spec: host: payment-service trafficPolicy: outlierDetection: consecutive5xxErrors: 5 ## open circuit after 5 consecutive 5xx errors interval: 30s baseEjectionTime: 30s ## eject for 30 seconds maxEjectionPercent: 50 ## eject at most 50% of endpoints ``` The senior insight: back pressure must be designed in. Retry logic without back pressure awareness (retrying immediately and infinitely) converts a partial outage into a total outage by amplifying load on an already struggling service. --- **Q27. How do you implement log aggregation at scale across multiple Kubernetes clusters?** The collection layer: Fluent Bit as a DaemonSet on every node. One pod per node reads container logs from /var/log/containers, enriches with Kubernetes metadata (pod name, namespace, labels, cluster name), and forwards to the aggregation layer: ```yaml # Fluent Bit config — add cluster identifier to every log line [FILTER] Name record_modifier Match * Record cluster ${CLUSTER_NAME} ## set via env var in DaemonSet Record environment production ``` The storage layer — Loki vs Elasticsearch: Loki indexes only labels (namespace, pod, service name), stores compressed log content in S3. Operationally simple. Queries with LogQL. For most engineering teams debugging incidents, this is sufficient. Elasticsearch indexes full log content — every word searchable. More powerful for compliance search and analytics across structured logs. More expensive to operate — Elasticsearch requires heap tuning, shard management, and careful index lifecycle management. Correlation requirement: every service must log a request ID in a consistent field name: ```json {"timestamp": "...", "service": "payment", "request_id": "req-abc-123", "level": "ERROR", "message": "..."} ``` With consistent request IDs, finding all logs across 50 services for a single user request is a single query: ``` {namespace="production"} | json | request_id="req-abc-123" ``` Retention and cost: error logs retain for 90 days. Info and debug for 14 days. Archive to S3 Glacier after retention period for compliance. This alone can reduce storage costs by 60-70%. Cross-cluster: each cluster runs its own Fluent Bit DaemonSet. All clusters forward to a central Loki. The cluster label in every log line lets you filter by cluster in Grafana. --- **Q28. What is the Kubernetes scheduler and how does it decide where to place a pod?** The scheduler watches for pods with no node assignment and selects the best node for each. It runs as a control plane component and uses a two-phase process: filtering then scoring. Filtering eliminates nodes that cannot host the pod. A node is filtered out if: * It does not have enough CPU or memory to satisfy the pod's requests * It has a taint that the pod does not tolerate * It does not match the pod's nodeSelector or nodeAffinity rules * It already has a pod with a conflicting podAntiAffinity rule * The PodTopologySpreadConstraint would be violated After filtering, the remaining nodes are scored. The scheduler uses multiple scoring functions including: how balanced the resource usage would be after placing the pod, whether the pod's preferred affinity rules are satisfied, whether image is already pulled on the node (avoids image pull delay). The highest-scoring node wins. The scheduler writes the node assignment to the pod object in etcd. The kubelet on that node picks it up and starts the container. ```yaml # Pod that will only schedule on nodes with SSD storage # and spread across zones spec: nodeSelector: storage-type: ssd topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: payment-service affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: payment-service topologyKey: kubernetes.io/hostname ## hard requirement: no two payment-service pods on the same node ``` Custom schedulers can be deployed for special workloads — GPU allocation, NUMA-aware placement for latency-sensitive services. Most workloads use the default scheduler with affinity rules and topology spread constraints. --- **Q29. What is AWS PrivateLink and when would you use it?** AWS PrivateLink allows you to expose a service running in your VPC to other VPCs — or to AWS services — without routing traffic through the internet or requiring VPC peering. Traditional VPC peering creates a full network connection between two VPCs. Any resource in VPC A can potentially reach any resource in VPC B on any port. This is broad. PrivateLink is narrow — you expose exactly one service on exactly one port, and the traffic never leaves the AWS network. Architecture: ``` Consumer VPC Provider VPC (your service) Client → Interface Endpoint → Network Load Balancer → Service (gets private IP in (your pods/EC2) consumer VPC) ``` The interface endpoint gets an IP address in the consumer's subnet. From the consumer's perspective, they are calling a private IP in their own VPC. They have no visibility into the provider's VPC — they cannot reach anything except the specific service you exposed. Use cases: Exposing your platform services to partner companies without VPC peering: a payments platform exposes its API over PrivateLink. Merchant systems connect to a private endpoint in their own VPC rather than hitting a public internet endpoint. Connecting to AWS services without internet gateway: many AWS services (S3, DynamoDB, ECR, Secrets Manager) support VPC Endpoints based on PrivateLink. EC2 instances in private subnets can access these services without a NAT Gateway — free for Gateway endpoints (S3, DynamoDB), hourly charge for Interface endpoints. Multi-account architectures: platform team's shared services (logging, monitoring, secrets management) exposed to application team accounts via PrivateLink without granting broad VPC peering access. The security benefit: traffic stays on the AWS backbone, never traverses the internet, and the consumer cannot reach beyond the exposed endpoint regardless of network configuration mistakes. --- **Q30. How do you handle a Kubernetes deployment that is stuck in a rollout — new pods are not becoming ready?** A stuck rollout means new pods are being created but never passing their readiness probe. The Deployment controller will not terminate old pods while the replacement is not ready — which is correct behaviour, but leaves you with mixed old and new versions running. Step 1 — Check the rollout status: ```bash kubectl rollout status deployment/payment-service -n production # Output: Waiting for deployment "payment-service" rollout to finish: 2 out of 5 new replicas have been updated... kubectl get pods -l app=payment-service -n production # Look for pods in Running but not Ready state, or in CrashLoopBackOff ``` Step 2 — Check why the new pods are not ready: ```bash # Describe the problematic pod kubectl describe pod <new-pod-name> -n production # Look at: Events section, Conditions section # Common issues shown here: # - Readiness probe failing (what endpoint, what error) # - Image pull errors (wrong tag, registry credentials) # - OOMKilled (memory limit too low for new version) # - Pending (insufficient resources on available nodes) # Check logs kubectl logs <new-pod-name> -n production # Application startup errors, missing config, failed dependency connections ``` Step 3 — Act based on root cause: Readiness probe failing because the app is not starting: check logs for startup errors — missing environment variable, changed config format, failed database migration. Image pull error: verify the image tag exists in the registry, verify imagePullSecrets are correctly configured. OOMKilled: the new version uses more memory. Either increase the memory limit or investigate the memory increase in the new version. Pending due to resources: the new version has higher CPU/memory requests than the old version and no node has capacity. Check `kubectl describe pod` for the "Insufficient cpu/memory" event. If the new version is fundamentally broken and you need to restore service immediately: ```bash # Rollback to previous known-good version kubectl rollout undo deployment/payment-service -n production # Watch the rollback complete kubectl rollout status deployment/payment-service -n production # Verify old version is fully restored kubectl get pods -l app=payment-service -n production ``` After rollback, fix the issue in staging, verify it passes health checks, and redeploy.
### Q46. Scenario - The 90-Day Plan You have just joined a 100-engineer company as a senior DevOps/platform engineer. It is your first week. Infrastructure is a mix of manually provisioned EC2, some Terraform (inconsistently applied), a Jenkins CI that nobody fully understands, and no formal on-call process. Deployments happen once a week through a shared Slack channel where someone posts "deploying service X" and manually SSHes into servers. Walk me through your first 90 days. ### What the interviewer is testing Prioritisation and change management under real constraints. This is a role-play scenario where showing up with a plan to migrate everything to Kubernetes in month one gets you failed, not hired. ### The right approach Week 1-2: Listen, do not fix. Talk to every engineer who touches infrastructure. Talk to the on-call engineers (if any). Find out what breaks most often, what they spend the most time on, what they wish they had. Write down everything you hear. Do not act on it yet. You are building a problem map, not a solution list. Specifically ask: "What was the last incident that was avoidable? What would have prevented it?" The answers tell you where the highest-value work is. Week 3-4: Establish baseline visibility. You cannot improve what you cannot see. Before changing anything, add observability. Install the CloudWatch agent on every EC2 instance if it is not there. Create a simple Grafana dashboard showing CPU, memory, disk, and network for the top 5 most critical services. Set up a single on-call alert per service: error rate above 2% for 5 minutes creates a PagerDuty incident. This is not your final observability architecture. It is minimum viable visibility. Month 2: Address the highest-risk deployment process. Manual SSH deploys to production are a single accident away from an outage and an audit failure. This is your first automation priority — not because Kubernetes is better than EC2, but because "one engineer fat-fingers a deployment and takes down production" is an unacceptable risk. Do not rebuild the CI/CD from scratch. Extend Jenkins (which people know) to do automated deployments instead of manual SSH. One small step: the deployment no longer requires a human to SSH. The pipeline does it. Validate in staging first. Month 3: First infrastructure as code win. Pick the simplest, least critical piece of infrastructure and convert it to Terraform. Not the production database. Not the EKS cluster you are planning. Something like the staging environment for one service. Run it through a PR review with the team. This does two things: it creates the first reviewed infrastructure change in the company's history, and it shows the team what the process feels like before you apply it to critical systems. At the end of 90 days, you should have: * Baseline observability across critical services * Automated deployments for at least one service (no more manual SSH) * One environment fully in Terraform * A list of the top 5 reliability risks, prioritised and shared with the team * Trust from the engineers because you listened before you changed things What you should NOT have done in 90 days: * Proposed a Kubernetes migration * Replaced Jenkins with a new CI system * Built a new observability stack from scratch * Created a "platform team" with a roadmap for the next year Those are 6-12 month projects. In your first 90 days, your job is to understand the system, reduce the highest-risk gaps, and build trust. Everything else follows from that. ### Q47. Scenario - The Architecture Decision Record Your team is deciding whether to adopt ArgoCD for GitOps deployments. Half the team wants to move forward. Half is skeptical — they are worried about the operational complexity. You have been asked to make the final recommendation. How do you make this decision and how do you communicate it? ### What the interviewer is testing Decision-making process under ambiguity, and the ability to communicate technical decisions to a mixed audience. ### The decision framework Reframe the question. It is not "should we use ArgoCD?" It is "what problem are we trying to solve, and is ArgoCD the right solution for that problem at our current scale?" Write an Architecture Decision Record (ADR). This is a short document (1-2 pages) that captures: the decision, the context, the options considered, the rationale, and the consequences. It is not just for this decision — it is a permanent record that future engineers can read to understand why the system is the way it is. ```markdown # ADR-017: Adopt ArgoCD for GitOps Deployments ### Status Proposed ### Context We deploy 15 services across 3 Kubernetes clusters. Currently, deployments are triggered by CI pipelines that kubectl apply directly to clusters. This requires CI to hold cluster credentials. Two incidents in Q3 were caused by CI pipeline misconfiguration accidentally deploying to the wrong cluster. ### Decision Drivers * Eliminate CI cluster credentials (security concern from last audit) * Improve deployment visibility (who deployed what, when) * Enable self-service deployment for development teams ### Options Considered 1. ArgoCD: pull-based GitOps, web UI, RBAC, broad adoption 2. Flux: pull-based GitOps, no web UI, lighter footprint 3. Continue with push-based CI: no migration cost, existing credential risk remains ### Decision Adopt ArgoCD. ### Rationale The credential security concern is the primary driver. ArgoCD eliminates CI cluster credentials — the highest-risk gap. The web UI reduces the barrier for development teams to see deployment status without kubectl access. Flux would also solve the credential problem but the web UI is a meaningful productivity benefit given our team composition. ### Consequences Positive: No CI cluster credentials. Deployment history in UI. Negative: ArgoCD is an additional system to operate. Learning curve of ~2 weeks for the team. Risk: If ArgoCD control plane is unavailable, deployments are blocked. Mitigation: Self-heal with 99.9% uptime SLO, deploy ArgoCD HA configuration. ``` How to communicate the decision: Send the ADR to the full team 48 hours before the meeting. Label it "proposed" — this signals you want input, not just buy-in. In the meeting: present the problem statement first, not the solution. "Two incidents last quarter were caused by CI cluster credentials. Our security audit flagged this as a high-risk gap. We need to address it." Get agreement on the problem before discussing the solution. Then present the options with their costs. "ArgoCD solves this, with these tradeoffs. Flux also solves this, with different tradeoffs. Staying as-is leaves the security risk open." Make the cost of each option concrete. The skeptical half of the team will have concerns. Address them directly in the document. "The concern about operational complexity is valid — ArgoCD is another system to maintain. The mitigation is deploying it in HA mode and treating it as a critical service with an SLO." Showing you took the concern seriously builds more buy-in than dismissing it. ### Q48. Scenario - The Cost Conversation Your CTO calls you into a meeting. The AWS bill jumped from 280k to 420k in one month — a 50% increase. She wants to know why and what you are doing about it. Before the meeting, pull the data from Cost Explorer broken down by service type, team, and environment. A 50% jump in one month almost always has a specific cause — a new service launched at large scale, data transfer from a new feature, a misconfigured auto-scaling policy, or a new database that was not rightsized. In the meeting: present the breakdown, not the total. "Here is where the increase came from. The new real-time recommendation service launched last month is responsible for 110k of the 140k increase. It is using on-demand EC2 instances that should be on Compute Savings Plans, and it is transferring data cross-region unnecessarily." Then present the plan: "In the next two weeks we can move recommendation to Savings Plans (saves 55k/month) and fix the cross-region data transfer (saves 20k/month). That recovers 75k of the 140k increase. The remaining 65k is justified — it is real traffic growth." The CTO does not want a technical explanation. She wants to know the number is understood, the cause is identified, and there is a plan. Lead with that. --- ### Q49. Scenario - The Thursday Deployment Decision It is Thursday afternoon. Your team is about to deploy a full rewrite of the authentication service that every other service depends on. Your engineering manager asks: deploy today or wait until Monday? Wait until Monday. Here is how you explain it: A Thursday afternoon deployment of a critical cross-cutting change has a narrow recovery window. If something goes wrong at 4 PM Thursday, you are debugging under pressure with a team tired from the week. If it takes more than 3-4 hours to stabilise, you are looking at a Friday evening incident. Monday gives you full team availability for the entire business day. What you say to the manager: "I recommend Monday. Not because the code is not ready — it is. Because the risk profile of a late-week auth rewrite is bad. If we absolutely must do it today, I want explicit agreement now on rollback criteria: if error rate on any dependent service crosses X% within 10 minutes of deploy, we auto-revert without discussion. Can we agree on that threshold and who makes the call?" This shows risk thinking, clear communication, and that you are not simply saying no — you are offering a concrete path forward with defined conditions. --- ### Q50. Scenario - The Platform Migration Proposal Your VP of Engineering asks you to evaluate moving the company from self-managed Kubernetes on EC2 to EKS. The company runs 40 services, has 8 engineers, and the platform team is 2 people including you. Present your recommendation. Start with what you are actually being asked: is this worth the migration cost, and if so, what does the migration look like? The honest assessment: self-managed Kubernetes has real ongoing cost — etcd backups, control plane upgrades, master node maintenance, certificate rotation. EKS removes all of that. For a 2-person platform team managing 40 services, that operational savings is significant. Your recommendation: migrate, but phased. Move non-production environments first. Run production on self-managed and EKS in parallel for 60 days. Migrate production one service group at a time starting with the lowest-risk services. Do not big-bang a 40-service migration. The question the VP will ask: "How long and how much engineering time?" Your honest answer: 3-4 months of part-time work for the platform team, roughly 60-80 engineer-hours total. The payback period is under 6 months given the reduction in on-call toil. What makes this a senior answer: you did not just say "EKS is better." You quantified the cost, proposed a risk-managed migration path, and gave the business a payback period. --- ### Q51. Scenario - The On-Call Burnout Three engineers on your team have complained separately that on-call is unsustainable. One is considering leaving. You are the senior engineer. Your manager asks you to fix it. What do you do? Diagnose before prescribing. Before changing the rotation or hiring, understand what is making on-call unsustainable. Pull the PagerDuty data: how many pages per week, at what hours, for which services, with what resolution time. Most on-call burnout comes from one of three things — too many noisy alerts, too few people in the rotation, or incidents that take too long to resolve because runbooks do not exist. If the data shows alert noise: audit every alert that fired in the last 30 days. Any alert with a false positive rate above 20% either gets fixed or gets deleted. Noisy alerts are worse than no alerts — they train engineers to ignore pages. If the data shows rotation is too thin: make the business case for either hiring or rotating more engineers through on-call. Present the data — "engineer X was paged 23 times in the last month, 14 of which were between 10 PM and 6 AM" — and let the cost speak. If incidents take too long: build runbooks for the top 10 most common incidents. A runbook that cuts a 45-minute investigation to a 10-minute procedure significantly changes on-call quality of life. What you say explicitly: "I will not fix this with a single change. I will instrument it, identify the top three causes, and address each one specifically. I will report back in 2 weeks with data and a plan." --- ### Q52. Scenario - The Security Audit Finding Your company undergoes a SOC 2 audit. The auditor flags three findings: EC2 instances have overly permissive IAM roles with admin access, CloudTrail is not enabled in two regions, and there are 14 IAM users with access keys that have not been rotated in over 365 days. You have 30 days to remediate before the follow-up audit. Prioritise and plan. Order of priority: the IAM access keys are the most urgent — long-lived credentials that have not been rotated are the most common source of AWS account compromise. Start here day one. Days 1-3: Audit every access key. For each one older than 90 days: identify what it is used for (CI pipeline, application, human user), rotate it immediately, and update every system using the old key. ```bash # Find all access keys and their age aws iam generate-credential-report aws iam get-credential-report --query 'Content' --output text | base64 -d | \ grep -v "^#" | awk -F',' '{print $1, $9, $11}' ``` Days 3-7: Enable CloudTrail in all regions. This is a one-hour task, not a 7-day task — the time buffer is for testing that logs are flowing correctly. Days 7-30: Remediate the overly permissive IAM roles. This is the most complex because you need to understand what each role actually needs before restricting it. Use IAM Access Analyzer to see what permissions are actually being used versus what is granted. Replace AdministratorAccess with scoped policies. What you tell the auditor at the follow-up: show the remediation evidence (CloudTrail showing when keys were rotated, screenshot of CloudTrail enabled in all regions, IAM policies updated with Access Analyzer evidence). Documentation of what you found and what you changed is as important as the fix itself. --- ### Q53. Scenario - The Incident Post-Mortem Resistance You ran a blameless post-mortem after a production incident that caused 2 hours of downtime. The engineer whose code caused the incident refuses to participate and says post-mortems are just a way to publicly blame people. Your manager asks you to handle it. The engineer's concern is legitimate even if their response is not. Post-mortems have a reputation for blame in many organisations. Address the concern directly. Talk to them privately first, not in the meeting: "I understand why you feel that way. A lot of post-mortems I have seen were blame sessions in disguise. This one is not going to be. Here is what we are actually going to discuss: the system conditions that made this possible, what we missed in our monitoring, and what we change in the process. Your code change is one data point in a chain of contributing factors. The goal is that this class of incident cannot happen again — not that someone gets punished." If they still resist: "I am not going to force you to participate. But I would like you to review the draft document and correct anything factually wrong before we share it. That is your right as someone directly involved." In the post-mortem itself: structure it so the first 10 minutes are the timeline only — no discussion of causes. The second section is contributing factors — plural, systemic, not individual. Code review missed it, staging did not catch it, the alert threshold was too high. The engineer's change is one item in a list of ten contributing factors. What you say at the end of the conversation: "I want you to know that running this post-mortem is about protecting the team from the next incident, not about you. If you see me running it in a way that feels like blame, tell me and I will stop." --- ### Q54. Scenario - The Inherited Disaster You take over ownership of a service from an engineer who left. The service handles payment reconciliation, runs on a single EC2 instance with no redundancy, has no monitoring, was last deployed 14 months ago, and has a deployment process documented as "SSH in and run deploy.sh." The service processes 50,000 transactions per day. What do you do? Do not touch it yet. Your first job is to understand it. A service that has been running untouched for 14 months is running. Touching it carelessly is more dangerous than the current state. Week 1: passive observation. Read every file on the server. Document what the service does, what it connects to, what cron jobs run. Look at the logs — what errors appear regularly, what is normal, what is abnormal. ```bash # What is running on this server? ps aux && cron -l && systemctl list-units --type=service # What does it connect to? netstat -an | grep ESTABLISHED # What does normal look like in the logs? tail -1000 /var/log/app/reconciliation.log | grep -v "INFO" | head -50 ``` Week 2: add monitoring without changing anything. Install CloudWatch agent. Set up one alert: if the service stops responding or if transaction processing stops for 30 minutes, page someone. You cannot fix what you cannot see. Week 3: write the real deployment runbook. Run a test deployment to a staging clone of the server. Document every step, every gotcha, every thing that could go wrong. The goal is that any engineer can deploy this service safely. Week 4: add a second instance behind a load balancer. Single instance means any hardware failure takes down payment reconciliation. This is the highest operational risk. Even a simple active-passive pair removes the single point of failure. What you tell your manager at week 1: "This service is stable but fragile. I am not going to rush changes. I am going to understand it first, add visibility, then incrementally reduce the risk. The worst thing I can do is deploy a fix that breaks something that has been working for 14 months." --- ### Q55. Scenario - The Build vs Buy Decision Under Pressure Your company is evaluating logging solutions. The data team wants Elasticsearch because they know it. The platform team (you) wants Loki because it is simpler to operate. Engineering leadership wants a decision in one week because the current solution is failing. How do you make this call and get alignment? Reframe the decision before making it. The question is not "which tool is better." The question is "which tool best fits our team's operational capacity and query requirements right now, knowing we may revisit in 18 months." Get the data in 48 hours, not a week: Who actually queries logs and what are they doing? Engineers debugging incidents need filtering by service and time. Analysts running ad-hoc searches need full-text indexing. These are different requirements. What is the operational cost of each? Elasticsearch requires heap tuning, shard management, and index lifecycle policies. One person who knows it can manage it, but if they leave, you have a knowledge gap. Loki is operationally simpler but has less query power. What does the immediate problem require? If the current solution is failing, the right answer is the one you can deploy and run reliably in the next 2 weeks — not the theoretically superior option that takes a month to configure correctly. How you get alignment: present both options with a decision matrix, not a recommendation. Show the data team the operational cost column. Show the platform team the query capability column. Let the matrix drive the decision, not the loudest voice in the room. What you say to engineering leadership: "I can have a recommendation with supporting data in 48 hours, not a week. The decision criteria are: time to deploy, operational overhead for the platform team, and whether it meets the top 3 query patterns our engineers actually use. I will interview 4 engineers today about their logging workflows and report back tomorrow." --- ### What Interviewers Actually Want at Senior Level Senior interviews test a completely different set of signals than mid-level. These are the behaviours that separate candidates who get offers from those who get strong technical feedback but no offer. You reason through tradeoffs out loud. Mid-level candidates give the answer. Senior candidates give the answer and then say "and the tradeoff is..." without being prompted. The interviewer is not just evaluating whether you know the answer — they are evaluating whether you think in tradeoffs. You know the limits of your knowledge. "I have not built this at that scale. Based on my experience at a smaller scale, my instinct is X, but I would want to validate that against how companies like Shopify have approached it before committing." This is a positive signal. Bluffing or giving vague answers to cover gaps is a strong negative signal. You connect technical decisions to business outcomes. "Adopting GitOps removes CI cluster credentials, which directly addresses our audit finding. It also gives us a deployment audit trail, which our Series B due diligence requires. The timing is actually good." Technical decisions do not exist in isolation. You can talk about a mistake clearly. The quality of your answer to "tell me about a mistake you made in production" is one of the highest-signal moments in a senior interview. Good: specific situation, specific actions, specific change you made afterward. Weak: vague story, blame on tooling or others, no specific change made. You include the humans in your technical answer. A migration plan that does not mention team training, change communication, or change resistance is incomplete. The interviewer knows you can build it — they want to know you can get a team to adopt it. Beyond technical answers, senior interviews test a completely different set of signals than mid-level. These are the specific behaviours that separate candidates who get offers from those who get strong technical feedback but no offer. ### You treat the interview as a conversation, not a test Mid-level candidates answer questions. Senior candidates engage with them. "That is an interesting constraint — can I ask what drove it? I want to make sure I am solving the right problem." This is not a trick. It reflects the actual difference in how senior engineers work — they interrogate requirements before designing solutions. ### You know the limits of your knowledge "I have not built a distributed tracing system at that scale. Based on my experience with Jaeger at smaller scale, my instinct is X, but I would want to read how companies like Shopify or LinkedIn have approached it before committing. What is your current thinking?" Saying you do not know something, with honesty about what you do know and what you would do to learn, is a positive signal at senior level. Bluffing or giving vague answers to avoid exposing gaps is a strong negative signal. ### You connect technical decisions to business outcomes "Adopting GitOps removes CI cluster credentials, which directly addresses our audit finding. It also gives us a deployment audit trail, which is a compliance requirement for our Series B due diligence. The timing is actually good." Technical decisions do not exist in a vacuum. Senior engineers understand why the business cares about the technical problem they are solving. ### You have made mistakes and can talk about them clearly Every senior engineer has made a production change that went wrong. The quality of the candidate's answer to "tell me about a mistake" is one of the highest-signal moments in a senior interview. What good looks like: "I made a Terraform change that inadvertently replaced a production database instead of modifying it. The state file showed replace, I did not catch it in the plan review because I was looking at the wrong section. We restored from backup with 47 minutes of data loss. We were lucky the RPO was low. After that, I added prevent_destroy to all stateful resources, changed our plan review process to require explicit sign-off on any destroy operation, and wrote the incident up as a runbook for the team." What weak looks like: "We had an incident with Terraform. I fixed it." Or worse: "It was actually not really my fault because the tooling did not make it clear."
**Q56. Tell me about a time you pushed back on a request from a senior stakeholder who wanted to move faster than was safe.** The Director of Product wanted to deploy a major payment gateway migration on a Friday afternoon before a long weekend. The migration touched the database schema, the API contract, and a third-party integration. I did not just say no — I built the case. I created a risk matrix on the spot: three things could go wrong, each with a probability and a recovery time. The most serious scenario had a 4-6 hour recovery with full team availability, but an 8-12 hour recovery on a holiday weekend with on-call only. I showed her the expected recovery time was 2.5x worse on Friday than Monday. She agreed to wait. The migration did have a schema issue requiring a 45-minute rollback — on a Monday with full team availability that was manageable. On a Friday before a long weekend it would have been a much worse conversation. What made it work: I did not just say no, I quantified the risk and showed the math. --- **Q57. Describe a production incident you caused. What happened and what did you change afterward?** I was migrating a service to a new VPC as part of a network restructuring. I had tested thoroughly in staging. What I had not tested was that the production database security group allowed the old VPC CIDR but not the new one. The service went dark immediately. Connection refused errors. Eight minutes of impact during peak traffic before I found the security group rule and added the new CIDR. Post-mortem identified three gaps: I had not verified security group rules against a production-identical configuration in staging, staging did not mirror production network topology, and I had no pre-flight connectivity checklist. Changes I made: built a network connectivity test suite — curl tests from the new subnet to every dependency before any cutover. Made it mandatory for all network changes. Added a quarterly task to verify staging matches production network topology. That checklist has caught three similar issues in the two years since. --- **Q58. Tell me about a time you had to influence a decision you had no authority over.** An application team decided to use a third-party logging library that sent logs directly to an external SaaS — bypassing our log aggregation pipeline. Security concern (PII going to an unreviewed vendor), cost concern (10x higher per GB than our Loki stack), and operational concern (we could no longer correlate logs across services). I had no authority over their library choices. So I built the case. I calculated the annual cost difference — a meaningful number. Got a security review flagging the PII concern. Then proposed an alternative: a thin wrapper library that gave them the simpler interface they actually wanted while keeping logs in our pipeline. Two teams adopted the wrapper. The original team switched after seeing the cost numbers. The lesson: influence without authority requires understanding what the other team actually wants and finding a way to give it to them while addressing your concerns. --- **Q59. Tell me about a time you disagreed with your team's technical consensus.** My team had consensus to migrate CI from Jenkins to GitHub Actions. I was the only one pushing back — not because GitHub Actions is bad, but because I had done an analysis nobody had quantified: 400 Jenkins pipelines with years of custom shared libraries, estimated 6-9 months of engineering time to migrate, both systems needing maintenance during the transition. I presented the numbers and proposed a hybrid approach: migrate new pipelines to GitHub Actions, keep existing Jenkins pipelines until services were deprecated or had a natural rewrite moment. No dedicated migration project. The team initially resisted — felt like I was defending Jenkins. I was defending our time. The hybrid approach was adopted. Two years later we had organically migrated 60% of pipelines without burning an explicit migration project. --- **Q60. Tell me about a time you gave critical feedback to a peer. How did you handle it?** A senior engineer merged a Terraform change that removed a prevent_destroy lifecycle block from a production database. The PR had passed review — it was buried in a large diff. I caught it in a code scan after merge, before it was applied. I went to them directly and privately. Showed them the change and explained what it would have done: the next terraform apply would have destroyed the production database. No harm done yet — but it needed to be understood. Their first reaction was defensive. I said: "Both of us missed it in review. That is a process problem. But I wanted to tell you directly because you are on-call next week when this gets applied." We fixed it together — restored the lifecycle block, added a CI check that fails if prevent_destroy is removed from critical resources. Turned a near-miss into a process improvement. The defensiveness faded once they understood I was not blaming them personally. --- **Q61. Describe a time you joined a new team and had to earn credibility quickly.** I joined a team mid-project during a high-stakes platform migration. They had been working on it for three months and had healthy skepticism of the new person showing up with ideas. My rule: listen first, contribute second, change things third. For two weeks I asked questions and took notes. I ran the operational tasks nobody wanted to do. Then I found a real technical issue — a gap where one service's database connections would not survive the network cutover. I wrote it up clearly with the fix and brought it to the tech lead privately. They fixed it, credited me in the team meeting, and from that point I had credibility. The path to credibility on a new team is not showing how much you know. It is showing you do the work, bring signal not noise, and do not make people look bad. --- **Q62. Tell me about a time you delivered a project with significantly fewer resources than expected.** We were approved to build a new observability platform with two engineers. A month in, one was pulled to a critical production issue that became a three-month assignment. I had to deliver the same scope alone. First I reset expectations on scope, not timeline. "The timeline stays. Here is what I can deliver by the original date with one engineer versus two." We agreed to drop two features from the initial release. Second I bought instead of built where buying was reasonable. I had planned to build a custom alerting UI — instead I configured PagerDuty's built-in integration. Three days instead of three weeks. The project shipped on time with the reduced scope. The two dropped features were added the following quarter. The stakeholders appreciated the early conversation about tradeoffs more than they would have appreciated a late delivery of the full scope. --- **Q63. Tell me about the most complex incident you led. Walk me through it.** A payment service was dropping 12% of transactions at 11 PM Friday. Three engineers had been debugging for 45 minutes with no root cause. First thing I did: establish an incident channel, assign roles. I took incident commander. One engineer on database layer, one on application layer, one maintaining a live timeline. My job was coordination, not debugging. Ten minutes in: database engineer reported query latency normal. Application engineer reported service was receiving and processing requests — but the downstream payment gateway was returning 422 errors on 12% of requests. I pulled in the integration engineer who had deployed a gateway API change the previous week. They confirmed a change had modified how we formatted the currency code field — from "INR" to "inr". The gateway was case-sensitive and had started rejecting lowercase values in a subset of transaction types — explaining the 12%. Fix: one-line config change. Total incident duration: 78 minutes. What I did right: assigned roles immediately, prevented duplicate debugging, pulled in domain knowledge rather than guessing, maintained the timeline document in real time. Post-mortem produced three process improvements: regression tests for integration field formats, a staging environment that mirrors the gateway's validation rules, and per-transaction-type error rate monitoring. --- **Q64. Describe a time you had to change your mind publicly after taking a strong technical position.** I had argued strongly for Kafka as our event streaming layer. I had experience with it, I trusted it, I made a confident case. Team adopted the recommendation. Three months into implementation, an engineer showed me an analysis that AWS Kinesis would meet all our requirements and reduce operational overhead by 60% — no brokers to manage, no replication config, no retention tuning. Total cost of ownership was lower. I had been wrong. Not about Kafka's capabilities — Kafka is excellent. But about whether those capabilities were needed for our use case. I had chosen the more powerful tool when the simpler one was sufficient. I said so explicitly in the next team meeting: "I was wrong about Kafka for this use case. Ravi's analysis shows Kinesis meets our requirements with much lower operational overhead. I am changing my recommendation." No hedging. No qualification. I was wrong and I said so. The team switched to Kinesis. The system has run reliably for two years with minimal operational attention. --- **Q65. Tell me about a time you saw a problem outside your direct responsibility and fixed it anyway.** Engineers on the data team were spending 3-4 hours per week running manual data pipeline verification scripts — checking that ETL jobs had completed and row counts matched between source and destination. Not my team's problem. I was on the platform team. But 3-4 hours per engineer per week is real engineering time. I had a conversation with the data team lead and asked if I could spend a sprint building a solution. I built a monitoring job that ran verification queries automatically after each ETL completion and posted results to Slack. Matched counts: green check. Mismatch: described the discrepancy. The manual process was replaced entirely. Total engineering time: 4 days. Time saved per engineer per week: 3-4 hours across 8 engineers — roughly one engineer-week per month recovered. The lesson: at senior level the boundary between "my problem" and "not my problem" is porous. If you see a problem you can fix and it is worth fixing, do it. --- **Q66. Tell me about a time you had to manage up — informing leadership about a problem they did not want to hear.** Six weeks before a major product launch I discovered our infrastructure could not handle projected load. Load testing showed database connection pool exhaustion at 40% of projected peak. The launch was tied to a marketing campaign — the timeline was fixed. I went to the VP of Engineering with the load testing results, the specific bottleneck, and the fix plan. The fix required two weeks of engineering work — deploying PgBouncer and tuning connection pool configurations. That fit within the six-week window if we started immediately. She was frustrated — nobody wants a problem six weeks before a launch. But she was not surprised by the candour. We started the fix that day. The launch happened on schedule. The system handled peak traffic without issue. What I did right: came with the problem and the solution in the same conversation. Provided specific numbers. Framed it as solvable. Bringing a problem without a plan creates anxiety. Bringing a problem with a plan creates a decision. --- **Q67. Describe a migration you led that was considered too risky to touch.** A legacy billing service: Java 8, single EC2 instance, no tests, the original engineer had left. It handled invoicing for all enterprise customers. Nobody would touch it. I started by understanding before touching. Two weeks reading code, tracing every code path, documenting what I found. I built a test harness capturing real production inputs and outputs — not unit tests, but integration tests that recorded actual API calls and verified responses matched before and after any change. First change: added a healthcheck endpoint. Just that. Deployed it. It worked. This built confidence that the deployment pipeline worked and changes could be made safely. Then logging. Then containerised it without changing any application code. Each step was reversible and verifiable against the test harness. Full modernisation took four months. No incidents. 80% test coverage at the end, running in Kubernetes, team could deploy without fear. The key was treating "understand it" and "change it" as separate phases. Most failed legacy migrations try to do both at once. --- **Q68. Tell me about a time you built something technically excellent that nobody adopted.** I built a comprehensive infrastructure cost dashboard. It pulled from Cost Explorer, broke down by team, service, and environment, compared actuals to budgets, highlighted top savings opportunities weekly. Technically excellent. Nobody used it. After a month I asked engineers why. Consistent answer: "It is not in my workflow. I check Slack, Jira, my IDE. I do not open a separate dashboard unless someone tells me to." I had built a solution to a problem I cared about and assumed others cared equally. They cared about cost in the abstract but not enough to add a new tool to their daily workflow. I rebuilt the delivery mechanism. Same data, but delivered as a weekly Slack summary per team lead showing only their team's costs and the top one or two savings opportunities. Actionable, in their existing workflow, specific to their context. Usage went from near zero to 80% of team leads checking it regularly. Two teams reduced monthly costs by 30% the following quarter. The lesson: a solution is only as good as its adoption. You have to design for the workflow people actually have, not the workflow you think they should have. --- **Q69. Describe a time you worked with a team that had a completely different technical philosophy.** I was asked to help the data science team improve their deployment process. Their philosophy: move fast, iterate, do not let process slow experiments. My philosophy: every production change needs testing, review, and rollback capability. These feel incompatible until you find the common ground. I spent time understanding what they were actually optimising for. The deployment friction was real — their model retraining pipeline went through the same review process as backend service changes. That was unnecessary friction for what was essentially a configuration update. I proposed a two-track process. Model artifacts (weights, configurations) could go through a lightweight fast-track — automated validation, no manual review, instant rollback by reverting the artifact version. Code changes to the serving infrastructure went through full review. This separated "experiment fast" work from "change production infrastructure" work. The data science team got the iteration speed they needed. The production serving infrastructure maintained the rigour needed to operate reliably. The lesson: when philosophies conflict, find what each side is actually optimising for. Usually they are optimising for different things in different contexts. --- **Q70. Tell me about a decision you made that you would make differently today.** Early in a platform role I made the call to build a custom internal metrics pipeline rather than adopting Prometheus. The reasoning at the time: our requirements were specific, the existing tools did not fit perfectly, we had strong engineering bandwidth. Two years later the custom pipeline was a maintenance burden that only two engineers fully understood, it had diverged significantly from industry patterns making onboarding hard, and when we wanted to adopt Grafana as our dashboarding layer the custom pipeline required months of integration work. If I were making that decision today I would start with Prometheus and extend it for our specific needs, rather than building from scratch. The cost of diverging from community standards compounds over time in ways that are hard to predict upfront — recruiting becomes harder, tooling integrations require custom work, documentation and training materials are written for the standard tools not yours. The principle I apply now: the bar for building custom over adopting community standard should be very high. The problem must be genuinely unsolvable by the standard tool, not just imperfectly solved. "We would have to configure it differently" is not a good reason to build from scratch. **Q71. Tell me about a time you had to build something under a hard deadline where cutting corners felt necessary. What did you cut and what did you refuse to cut?** We were building a new deployment pipeline for a product launch in 3 weeks. Full automated testing, canary deployment, automatic rollback, and observability integration were on the plan. It became clear by week 1 that all four were not achievable in time. I made an explicit list of what we would cut and what we would not. Non-negotiable: automatic rollback on error rate spike — this was the safety net if anything went wrong on launch day. Non-negotiable: production monitoring and alerting — we could not fly blind on launch day. What we cut: the canary deployment (we would deploy to 100% and rely on rollback), and the full automated test suite (we kept smoke tests only and accepted manual QA for this launch). I documented every cut explicitly in a ticket labelled "post-launch hardening" with the rationale for each decision. Three weeks after launch, we completed every item on that list. What I would say in the interview: "Cutting corners without documenting them is how technical debt becomes invisible. Every shortcut I take, I write down as a known risk, assign it a deadline, and make sure my manager knows it exists. That way the decision to cut is explicit and reversible, not an accident." --- **Q72. Describe a situation where you had to coordinate a response across multiple teams during a major incident. What was your role and what did you do well or poorly?** We had a cascading failure that touched the payments team, the platform team, and the data team simultaneously. The payments API was returning errors, the platform team's Kafka cluster was showing lag, and the data team's reconciliation jobs were failing because they depend on Kafka. I took incident command. The first thing I did — which I should have done faster — was create a dedicated incident channel and assign each team a specific investigation track. I waited about 4 minutes before doing this because I was trying to diagnose first. That was a mistake. The teams were duplicating effort and stepping on each other's investigation. Once I created the channel and assigned tracks: payments team on the API errors, platform team on Kafka, data team on hold until we understood the root cause. Within 8 minutes we had it: a Kafka broker had run out of disk space, which caused producer timeouts, which caused the payments API to fail on its async event publishing path. Fix was expanding the broker volume. Total incident time: 34 minutes. If I had assigned roles at minute 1 instead of minute 4, probably 24 minutes. What I would say about what I did poorly: "I let my instinct to diagnose myself override the incident commander's job, which is to coordinate, not debug. The lesson I carry from this: as incident commander, my job is to get the right people on the right problem as fast as possible. My individual debugging is often the least valuable use of my time in the room." --- **Q73. Tell me about a time you had to kill a project or initiative you had championed. How did you make the decision and communicate it?** I had spent 6 weeks building a case for migrating our secrets management from Kubernetes Secrets to HashiCorp Vault. Got approval, started implementation. Eight weeks into the 12-week project, two things became clear: the migration was more complex than estimated because of how many services had secrets baked into their deployment configs, and the company had just signed a contract for AWS Secrets Manager as part of a broader enterprise AWS agreement. I had to recommend stopping my own project and pivoting to a solution I had not championed. I wrote a short document: here is what we have built so far (portable, nothing wasted), here is why the original estimate was wrong, here is why AWS Secrets Manager now makes more sense given the contract, and here is the migration path from what we built to the new approach. I presented it to my manager as a recommendation, not a confession. "Here is the situation, here is the analysis, here is what I recommend we do." Not "I was wrong and I am sorry." The key thing: I did not drag the project out or try to justify continuing it because I had invested 8 weeks. Sunk cost is sunk. The question was which path is right from today forward. What I would say in the interview: "The hardest part was telling the team who had been working with me. I did it honestly — here is why I recommended this, here is what changed, here is what we learned that we will use in the AWS Secrets Manager implementation. Nobody felt their work was wasted because I explained how it informed the next decision." --- **Q74. How do you decide when a problem is worth escalating versus solving yourself?** The framework I use is: how much longer will it take me to solve this alone versus how long will it take to get help, and what is the cost of the delay either way. If I have been stuck on something for more than 30 minutes and I have a specific question — not a vague "I do not understand this" but a specific "I have tried X and Y and I think the issue is Z but I cannot confirm it because of this specific gap" — I ask. Specific questions get specific answers. Vague questions waste everyone's time including mine. In production incidents, my threshold is much lower. If I have been investigating for 10 minutes and I am not converging on a hypothesis, I pull in a second person. The cost of a second pair of eyes on a production incident is almost always worth it. For escalating to management: I escalate when the problem has business impact I cannot resolve with technical means, when I need a decision that is above my authority (spending money, changing a process that affects other teams), or when I have been going in circles for long enough that I need external input on whether I am solving the right problem. What I do not do: escalate every hard problem to avoid discomfort, or stay silent on a problem that is getting worse because I am afraid to admit I do not have it under control. Both are failure modes I have seen damage team trust. --- **Q75. Where do you see the gap between senior and staff engineer level? What are you actively working on to close it?** The gap I see most clearly is scope. A senior engineer owns a system or a service. A staff engineer influences how systems are built across multiple teams, often without direct authority over the engineers doing the building. The technical gap is real but smaller than people think. Staff engineers are not necessarily better at Kubernetes than senior engineers. What they are better at is translating technical decisions into organisational change — writing the document that gets three teams to adopt a new standard, designing the platform that makes the right choice the easy choice for every team, spotting the architectural decision that will constrain the company three years from now. What I am working on specifically: I have been deliberately taking on cross-team projects rather than staying in my team's lane. Last quarter I led the evaluation of our logging infrastructure which required building consensus across the data team, the platform team, and three application teams. I ran the process, wrote the recommendation, and got alignment without having authority over any of the teams involved. That is the muscle I am building. The other gap I am aware of: written communication. Staff-level influence happens mostly through documents — RFCs, ADRs, post-mortems, strategy memos. I have been writing more of these and asking for feedback on them specifically from engineers I respect who are already at staff level.
Senior DevOps Engineer (5-8 years) | Company Type | Range | |:-------------|:------| | Service company | ₹20L - ₹32L | | Mid-size product startup | ₹28L - ₹42L | | Well-funded growth startup | ₹35L - ₹50L | | Large product company (PhonePe, Atlassian India tier) | ₹45L - ₹65L | | FAANG / hyperscaler India offices | ₹55L - ₹90L+ | The difference between ₹32L and ₹55L for the same years of experience is almost entirely explained by two things: whether you can demonstrate system design ownership at scale, and whether you can show you have driven reliability or cost improvements with measurable outcomes. Both of those are things this module has prepared you to talk about.
You have been in production. You have debugged things at 2 AM. You have written Terraform, built pipelines, and owned a ...
These are table stakes. If you need to look any of these up, go back to the relevant module first. Kubernetes - Platform...
System Design and Architecture Designing Scalable Infrastructure from First Principles The Question You are joining a fu...
Q46. Scenario - The 90-Day Plan You have just joined a 100-engineer company as a senior DevOps/platform engineer. It is ...
Q56. Tell me about a time you pushed back on a request from a senior stakeholder who wanted to move faster than was safe...
Senior DevOps Engineer (5-8 years) Company Type Range Service company ₹20L - ₹32L Mid-size product startup ₹28L - ₹42L W...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.