Every other capstone taught you to build things. Capstone 1 built an application. Capstone 2 provisioned infrastructure. Capstone 3 created a delivery platform. Capstone 4 built a developer portal. All of them assumed things would go correctly. This one does not. This capstone is the difference between an engineer who has studied platform engineering and an engineer who operates it. In production, things break. Not occasionally — regularly. Memory leaks. Bad deployments. Network policies that are too strict. Databases that run out of connections. Cost spikes from a forgotten load test. You do not get to choose when these happen or what combination arrives together. The only way to be ready is to have broken things intentionally, in a controlled environment, and learned how to find and fix each failure before it finds you at 2 AM. **Chaos engineering** is the practice of deliberately injecting failures into a system to discover weaknesses before users do. Netflix invented it with Chaos Monkey — a service that randomly kills production pods to ensure the system can survive node failures. Google uses it. Razorpay runs chaos experiments before major sale events to verify the payment system holds under pressure. This capstone applies the same discipline. **How each scenario is structured:** * **Setup** — get the system into a known good state * **Inject** — deliberately break something specific * **Detect** — use observability tools to find the problem * **Fix** — resolve the issue with the correct approach * **Prevent** — add a safeguard so it cannot happen again undetected Run every scenario on the staging cluster from Capstone 2. Never run chaos experiments on production without automated rollback, executive approval, and a full on-call team standing by. ---
### Setup Before injecting anything, verify the system is in a known good state. ```bash ## Verify all pods are running and healthy kubectl get pods -n swiggy-clone-staging ## All pods should show STATUS: Running, RESTARTS: 0 ## Verify the API is responding curl -s http://swiggy-clone-staging.your-cluster.example/health/ready ## Should return: {"status": "ok"} echo "✅ System healthy — ready to inject failure" ``` ### Inject A CrashLoopBackOff happens when a pod starts, crashes immediately, and Kubernetes keeps restarting it in a loop. This usually means the application cannot start — often because a required environment variable or dependency is missing. ```bash ## Inject a bad environment variable — clear the database host ## The backend expects DB_HOST to be set — clearing it causes it to crash on startup kubectl set env deployment/backend DB_HOST="" -n swiggy-clone-staging ## Watch what happens kubectl get pods -n swiggy-clone-staging --watch ## Within 30 seconds you will see: ## NAME READY STATUS RESTARTS ## backend-xxx 0/1 CrashLoopBackOff 3 ## backend-xxx 0/1 Error 3 ## The restart count keeps climbing ``` ### Detect ```bash ## Step 1: Confirm the CrashLoopBackOff kubectl get pods -n swiggy-clone-staging ## STATUS column shows CrashLoopBackOff — the pod is crashing repeatedly ## Step 2: Read the current crash logs kubectl logs deployment/backend -n swiggy-clone-staging ## May be empty if the crash is very fast — use --previous instead ## Step 3: Read logs from the PREVIOUS container run ## --previous reads logs from the last container that crashed kubectl logs deployment/backend -n swiggy-clone-staging --previous ## Look for: "DB_HOST is required", "Cannot connect to database", ## "Connection refused", or any startup error from your application code ## Step 4: Describe the pod for more detail kubectl describe pod -l app=backend -n swiggy-clone-staging | tail -30 ## Look for the Events section: ## Warning BackOff 2m kubelet Back-off restarting failed container ## Step 5: Check if it is an environment variable problem kubectl exec deployment/backend -n swiggy-clone-staging -- env | grep DB ## If the pod can start long enough to exec, you will see DB_HOST is empty ## If it cannot, this command fails — which itself confirms the startup crash ``` > 📌 **Remember:** `kubectl logs` without `--previous` shows the current container's logs. If a pod crashes immediately, there are no current logs. Always try `--previous` first for CrashLoopBackOff. ### Fix ```bash ## Restore the correct database host kubectl set env deployment/backend \ DB_HOST=postgres-service \ -n swiggy-clone-staging ## Watch the pods recover kubectl rollout status deployment/backend -n swiggy-clone-staging ## Output: deployment "backend" successfully rolled out ## Verify healthy kubectl get pods -n swiggy-clone-staging ## STATUS should return to Running, RESTARTS will show the crash count ## but no new restarts should be occurring ``` ### Prevent The application should validate all required environment variables at startup and crash immediately with a clear message if any are missing. This makes CrashLoopBackOff self-diagnosing — the log message tells you exactly what is wrong. ```javascript // src/startup-validation.js // Add this as the first thing your application runs const REQUIRED_ENV_VARS = [ 'DB_HOST', 'DB_PORT', 'DB_NAME', 'DB_USER', 'DB_PASSWORD', 'REDIS_HOST', 'PORT', ]; function validateEnvironment() { const missing = REQUIRED_ENV_VARS.filter(key => !process.env[key]); if (missing.length > 0) { // This message appears in kubectl logs --previous // Making it specific means the engineer reading it knows exactly what to fix console.error(`FATAL: Missing required environment variables: ${missing.join(', ')}`); console.error('The application cannot start without these values.'); console.error('Check your Kubernetes Secret and Deployment environment config.'); process.exit(1); // non-zero exit causes Kubernetes to count it as a crash } } validateEnvironment(); // call this before any other application code ``` ---
### What OOMKill is When a Kubernetes pod exceeds its memory limit, the Linux kernel's OOM (Out of Memory) killer terminates the process. Kubernetes then restarts it. Unlike a crash from a bug, the pod did not fail — it was killed by the system for consuming too many resources. OOMKill is common in two situations: the memory limit is set too low for the actual workload, or the application has a memory leak that causes usage to grow over time until it hits the ceiling. ### Setup ```bash ## Set a very low memory limit — 32Mi is too small for a Node.js application kubectl set resources deployment/backend \ -c backend \ --limits=memory=32Mi \ -n swiggy-clone-staging ## Wait for pods to restart with new limits kubectl rollout status deployment/backend -n swiggy-clone-staging ``` ### Inject ```bash ## Send requests that cause memory usage to spike ## Install 'hey' load testing tool if not installed go install github.com/rakyll/hey@latest ## Send 500 concurrent requests — this drives memory usage up hey -n 1000 -c 50 http://swiggy-clone-staging.your-cluster.example/api/menu ## You do not need to wait for this to finish — watch the pods in another terminal ``` ### Detect ```bash ## Terminal 1: Watch pods in real time kubectl get pods -n swiggy-clone-staging --watch ## You will see the pod status change: ## backend-xxx 1/1 Running 0 → 0/1 OOMKilled 1 ## Terminal 2: Describe the pod after it crashes kubectl describe pod -l app=backend -n swiggy-clone-staging | grep -A 10 "Last State" ## Output: ## Last State: Terminated ## Reason: OOMKilled ← this is the key indicator ## Exit Code: 137 ← exit code 137 = killed by signal 9 (SIGKILL) ## Check Prometheus for memory usage ## Run this PromQL query in Grafana or the Prometheus UI: ## container_memory_working_set_bytes{namespace="swiggy-clone-staging", container="backend"} ## You will see the spike just before the OOMKill ## Check Grafana for the memory spike ## Dashboard: Kubernetes / Compute Resources / Pod ## Filter: namespace=swiggy-clone-staging, pod=backend-xxx ## The memory graph will show a sharp climb to 32Mi then drop to zero (restart) ``` > 📌 **Remember:** Exit code 137 always means OOMKill. Exit code 1 means the application itself exited with an error. Exit code 0 means the application exited cleanly (probably a job that finished). These codes are your first diagnostic signal. ### Fix ```bash ## Step 1: Find out how much memory the backend actually uses under normal load ## Query Prometheus for P95 memory usage over the last 24 hours ## PromQL: quantile_over_time(0.95, ## container_memory_working_set_bytes{ ## namespace="swiggy-clone-staging", ## container="backend" ## }[24h]) ## If result is ~180Mi, set limit to 256Mi (1.5x headroom is a good rule) ## Step 2: Set a sensible memory limit based on actual usage kubectl set resources deployment/backend \ -c backend \ --requests=memory=128Mi \ --limits=memory=512Mi \ -n swiggy-clone-staging ## Step 3: Verify pods restart cleanly kubectl rollout status deployment/backend -n swiggy-clone-staging kubectl get pods -n swiggy-clone-staging ## RESTARTS should not be climbing anymore ``` ### Prevent ```bash ## Install Vertical Pod Autoscaler (VPA) from the FinOps module ## VPA watches actual CPU/memory usage and recommends correct limits kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vertical-pod-autoscaler.yaml ## Create a VPA object for the backend in recommendation-only mode ## Recommendation mode: suggests values but does not change them automatically ## This is the safest starting point — you review recommendations before applying ``` ```yaml ## k8s/vpa-backend.yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: backend-vpa namespace: swiggy-clone-staging spec: targetRef: apiVersion: apps/v1 kind: Deployment name: backend updatePolicy: updateMode: "Off" ## Off = recommend only, do not change pods automatically ``` ```bash kubectl apply -f k8s/vpa-backend.yaml ## After 24 hours, check the VPA recommendations kubectl describe vpa backend-vpa -n swiggy-clone-staging | grep -A 10 Recommendation ## Output: ## Lower Bound: cpu: 50m, memory: 100Mi ## Target: cpu: 120m, memory: 180Mi ← set your limits based on Target ## Upper Bound: cpu: 300m, memory: 400Mi ## Uncapped Target: cpu: 120m, memory: 180Mi ``` > 🔴 **Common Mistake:** Sizing memory limits based on average usage. If average is 100Mi but P95 is 180Mi, setting a limit of 120Mi will OOMKill during traffic spikes. Always size limits from the P95 or P99 percentile, not the average. ---
### Setup ```bash ## Verify a clean starting state kubectl get rollout backend -n swiggy-clone-staging ## STATUS: Healthy, STABLE: backend:v1.5.0 or similar ## Confirm the application is responding curl -s http://swiggy-clone-staging.your-cluster.example/health/ready ## Returns: {"status": "ok"} ``` ### Inject ```bash ## Deploy an image tag that does not exist in the registry ## This simulates a CI pipeline that accidentally pushed a wrong tag name kubectl set image rollout/backend \ backend=your-account.dkr.ecr.ap-south-1.amazonaws.com/swiggy-backend:nonexistent-broken \ -n swiggy-clone-staging ## Watch what happens kubectl get pods -n swiggy-clone-staging --watch ## STATUS changes to: ImagePullBackOff ## The rollout is stuck — new pods cannot start, old pods are still running ``` ### Detect ```bash ## Check the pod status kubectl get pods -n swiggy-clone-staging ## backend-new-xxx 0/1 ImagePullBackOff 0 ## Describe the failing pod kubectl describe pod -l app=backend -n swiggy-clone-staging | grep -A 5 "Events:" ## Events: ## Warning Failed 30s kubelet Failed to pull image: not found ## Check ArgoCD — it shows the rollout health argocd app get swiggy-clone-staging ## Health Status: Degraded ## This tells you ArgoCD detected the rollout did not complete successfully ## Check the Argo Rollout status directly kubectl argo rollouts get rollout backend -n swiggy-clone-staging ## Status: ✖ Degraded ## Strategy: Canary ## Step: 0/7 ## The canary step shows the broken image tag — the old stable version ## is still serving 100% of traffic because the canary never passed Step 1 ``` ### Fix ```bash ## Option 1: ArgoCD rollback via the UI ## Open the ArgoCD UI → swiggy-clone-staging → HISTORY AND ROLLBACK ## Click Rollback on the previous healthy revision → Confirm ## Option 2: kubectl rollout undo (fast, terminal-only) kubectl argo rollouts undo rollout/backend -n swiggy-clone-staging ## This immediately restores the previous stable image tag ## Option 3: Fix forward — deploy the correct image tag kubectl set image rollout/backend \ backend=your-account.dkr.ecr.ap-south-1.amazonaws.com/swiggy-backend:v1.5.0 \ -n swiggy-clone-staging ## Verify recovery kubectl argo rollouts get rollout backend -n swiggy-clone-staging ## Status: Healthy ``` ### Prevent Add image tag validation to the CI pipeline so a tag that does not exist in the registry cannot be deployed. ```yaml ## .github/workflows/deploy.yml ## Add this validation step before updating the Kubernetes manifest - name: Validate image exists in ECR before deploying run: | ## Check if the image tag exists in ECR ## This fails the pipeline if the image was not pushed successfully aws ecr describe-images \ --repository-name swiggy-backend \ --image-ids imageTag=${IMAGE_TAG} \ --region ap-south-1 echo "✅ Image ${IMAGE_TAG} confirmed in ECR — safe to deploy" ``` ---
### What connection exhaustion is Every PostgreSQL database has a maximum number of simultaneous connections (`max_connections`). The default is 100. If you have 20 pods each opening 10 connections, you have 200 connections — exceeding the limit. New connection attempts fail with `FATAL: remaining connection slots are reserved`. Every API request that needs the database returns a 500 error. Without a **connection pooler** like PgBouncer, every pod holds open database connections for its entire lifetime. As you scale out, connection count grows linearly. With PgBouncer, pods connect to PgBouncer (which can handle thousands of connections) and PgBouncer maintains a smaller pool to the actual database. ### Setup ```bash ## Check current connection count — baseline before injection kubectl exec -n swiggy-clone-staging deployment/postgres -- \ psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;" ## Note this number — likely 5-10 with a small deployment ``` ### Inject ```bash ## Remove connection pool limits from the backend and scale to 20 replicas ## Each replica will open a new pool of connections to PostgreSQL kubectl set env deployment/backend \ DB_POOL_SIZE=20 \ -n swiggy-clone-staging kubectl scale deployment backend --replicas=20 -n swiggy-clone-staging ## Watch connection count climb kubectl exec -n swiggy-clone-staging deployment/postgres -- \ psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;" ## Runs the query every 5 seconds — watch the count climb toward 100 ## Send requests to trigger connection failures hey -n 500 -c 100 http://swiggy-clone-staging.your-cluster.example/api/menu ## API will start returning 500 errors when connection limit is hit ``` ### Detect ```bash ## Check PostgreSQL logs for connection errors kubectl logs deployment/postgres -n swiggy-clone-staging | \ grep -i "connection\|too many\|remaining" ## Look for: FATAL: remaining connection slots are reserved for non-replication ## Look for: FATAL: sorry, too many clients already ## Check the API error rate in Prometheus ## PromQL: rate(http_requests_total{status=~"5..",app="backend"}[5m]) ## You will see a spike in 5xx errors coinciding with scale-out ## Check connection count by application kubectl exec -n swiggy-clone-staging deployment/postgres -- \ psql -U postgres -c " SELECT application_name, count(*) FROM pg_stat_activity GROUP BY application_name ORDER BY count DESC;" ## This shows which application is consuming connections ``` ### Fix ```bash ## Step 1: Immediately scale back down to stop the bleeding kubectl scale deployment backend --replicas=3 -n swiggy-clone-staging ## Step 2: Restore sensible connection pool size kubectl set env deployment/backend \ DB_POOL_SIZE=5 \ -n swiggy-clone-staging ## Step 3: Deploy PgBouncer as a connection pooler ``` ```yaml ## k8s/pgbouncer.yaml ## PgBouncer sits between the application and PostgreSQL ## Pods connect to PgBouncer instead of PostgreSQL directly apiVersion: apps/v1 kind: Deployment metadata: name: pgbouncer namespace: swiggy-clone-staging spec: replicas: 1 selector: matchLabels: app: pgbouncer template: metadata: labels: app: pgbouncer spec: containers: - name: pgbouncer image: pgbouncer/pgbouncer:1.21.0 ports: - containerPort: 5432 env: - name: DATABASES_HOST value: postgres-service ## the real PostgreSQL service name - name: DATABASES_PORT value: "5432" - name: DATABASES_DBNAME value: swiggy - name: PGBOUNCER_POOL_MODE value: transaction ## transaction pooling: most efficient - name: PGBOUNCER_MAX_CLIENT_CONN value: "1000" ## PgBouncer accepts up to 1000 connections - name: PGBOUNCER_DEFAULT_POOL_SIZE value: "20" ## PgBouncer keeps only 20 connections to PostgreSQL resources: requests: cpu: "50m" memory: "32Mi" limits: cpu: "200m" memory: "64Mi" --- apiVersion: v1 kind: Service metadata: name: pgbouncer-service namespace: swiggy-clone-staging spec: selector: app: pgbouncer ports: - port: 5432 targetPort: 5432 ``` ```bash kubectl apply -f k8s/pgbouncer.yaml ## Update the backend to connect to PgBouncer instead of PostgreSQL directly kubectl set env deployment/backend \ DB_HOST=pgbouncer-service \ -n swiggy-clone-staging ## Verify connection count dropped kubectl exec -n swiggy-clone-staging deployment/postgres -- \ psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;" ## Should now show ~20 connections (PgBouncer pool) regardless of backend replica count ``` ### Prevent ```yaml ## k8s/resource-quota.yaml ## ResourceQuota limits the maximum number of replicas in a namespace ## This prevents accidental scale-out that exhausts database connections apiVersion: v1 kind: ResourceQuota metadata: name: namespace-limits namespace: swiggy-clone-staging spec: hard: ## Maximum 10 pods in this namespace — prevents runaway scaling pods: "10" ## Maximum CPU and memory across all pods in the namespace requests.cpu: "4" requests.memory: 4Gi limits.cpu: "8" limits.memory: 8Gi ``` ---
### Setup ```bash ## Verify Kubecost is installed (from the FinOps module) kubectl get pods -n kubecost ## Should show kubecost-cost-analyzer Running ## Open the Kubecost dashboard and note the current daily cost ## http://localhost:9090 (after port-forward) kubectl port-forward -n kubecost deployment/kubecost-cost-analyzer 9090:9090 & ## Note the current cost for swiggy-clone-staging namespace as a baseline ``` ### Inject ```bash ## Create a Deployment with no resource requests and scale it to 50 replicas ## Without resource requests, Kubernetes cannot schedule efficiently ## Pods claim no resources on paper but consume actual CPU and memory ## Kubecost cannot correctly attribute cost — and the waste is enormous kubectl create deployment cost-hog \ --image=nginx:latest \ -n swiggy-clone-staging ## Scale to 50 replicas immediately kubectl scale deployment cost-hog --replicas=50 -n swiggy-clone-staging ## Watch nodes become pressured kubectl describe nodes | grep -A 5 "Allocated resources" ## CPU and memory allocation will spike significantly ``` ### Detect ```bash ## Open Kubecost dashboard at http://localhost:9090 ## Navigate to: Allocations → Namespace → swiggy-clone-staging ## You will see a sudden spike in cost for the cost-hog deployment ## The efficiency score will drop because there are no resource requests set ## Check from the terminal kubectl get deployment cost-hog -n swiggy-clone-staging ## READY column will show 50/50 — 50 pods consuming untracked resources ## Check cluster capacity impact kubectl top nodes ## CPU% and Memory% on nodes will have jumped significantly ## Prometheus alert would have fired (if configured from FinOps module): ## KubecostBudgetExceeded: namespace swiggy-clone-staging exceeded daily budget ``` ### Fix ```bash ## Delete the runaway Deployment immediately kubectl delete deployment cost-hog -n swiggy-clone-staging ## Verify cost returns to baseline kubectl top nodes ## CPU% and Memory% should return to previous levels within 2 minutes ## Check Kubecost shows the cost dropping back to baseline ## This may take up to 15 minutes for Kubecost to reflect the change ``` ### Prevent ```yaml ## k8s/limit-range.yaml ## LimitRange sets default resource requests on pods that do not specify them ## It also sets a minimum — pods cannot be created without at least these resources apiVersion: v1 kind: LimitRange metadata: name: default-limits namespace: swiggy-clone-staging spec: limits: - type: Container ## If a pod does not specify requests, these defaults are applied automatically defaultRequest: cpu: "50m" memory: "64Mi" ## If a pod does not specify limits, these defaults are applied default: cpu: "500m" memory: "512Mi" ## Minimum allowed values — a pod cannot request less than these min: cpu: "10m" memory: "32Mi" ## Maximum allowed values — a pod cannot request more than these max: cpu: "2" memory: "4Gi" ``` ```yaml ## k8s/kyverno-require-resources.yaml ## Kyverno policy: blocks any Deployment that does not have resource requests ## This is enforced at admission time — the Deployment cannot be created apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-resource-requests spec: validationFailureAction: enforce ## block, do not just warn rules: - name: require-requests match: any: - resources: kinds: - Deployment validate: message: "Resource requests (CPU and memory) are required on all containers. Set spec.containers[].resources.requests." pattern: spec: template: spec: containers: - resources: requests: memory: "?*" ## ?* means: must be present and non-empty cpu: "?*" ``` ---
Every other capstone taught you to build things. Capstone 1 built an application. Capstone 2 provisioned infrastructure....
Setup Before injecting anything, verify the system is in a known good state. Inject A CrashLoopBackOff happens when a po...
What OOMKill is When a Kubernetes pod exceeds its memory limit, the Linux kernel's OOM (Out of Memory) killer terminates...
Setup Inject Detect Fix Prevent Add image tag validation to the CI pipeline so a tag that does not exist in the registry...
What connection exhaustion is Every PostgreSQL database has a maximum number of simultaneous connections (maxconnections...
Setup Inject Detect Fix Prevent ---...
Setup Inject Detect Fix Prevent ---...
What an SLO is An SLO (Service Level Objective) is a target reliability level for a service. It answers the question: ho...
---...
❌ Running chaos experiments in production before staging 💥 An engineer decides to test how the production cluster handl...
The production debugging flowchart — the exact commands for every common failure. Pod not running Service returning erro...
Five capstones. Each one building on the last. Each one mapping directly to how production Platform Engineering teams wo...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.