Production Readiness Challenge

The final test. Take a running application, inject real production failures - pod crashes, OOMKills, bad deployments, cost spikes, network issues - and learn to diagnose and fix each one.

Related Concepts & TermsDeployment Grafana Prometheus

Domains & Technologies

Domains

CAPSTONESREDEBUGGINGOBSERVABILITYCHAOSPRODUCTION-READINESSINCIDENT-RESPONSESLO

Technologies

PROMETHEUSGRAFANA

Blueprint Walkthrough

Before You Start — Read This First

Every other capstone taught you to build things. Capstone 1 built an application. Capstone 2 provisioned infrastructure. Capstone 3 created a delivery platform. Capstone 4 built a developer portal. All of them assumed things would go correctly.

This one does not.

This capstone is the difference between an engineer who has studied platform engineering and an engineer who operates it. In production, things break. Not occasionally — regularly. Memory leaks. Bad deployments. Network policies that are too strict. Databases that run out of connections. Cost spikes from a forgotten load test. You do not get to choose when these happen or what combination arrives together.

The only way to be ready is to have broken things intentionally, in a controlled environment, and learned how to find and fix each failure before it finds you at 2 AM.

Chaos engineering is the practice of deliberately injecting failures into a system to discover weaknesses before users do. Netflix invented it with Chaos Monkey — a service that randomly kills production pods to ensure the system can survive node failures. Google uses it. Razorpay runs chaos experiments before major sale events to verify the payment system holds under pressure. This capstone applies the same discipline.

How each scenario is structured:

Setup — get the system into a known good state
Inject — deliberately break something specific
Detect — use observability tools to find the problem
Fix — resolve the issue with the correct approach
Prevent — add a safeguard so it cannot happen again undetected

Run every scenario on the staging cluster from Capstone 2. Never run chaos experiments on production without automated rollback, executive approval, and a full on-call team standing by.

Scenario 1 — Pod CrashLoopBackOff

Setup

Before injecting anything, verify the system is in a known good state.

Bash

## Verify all pods are running and healthy
kubectl get pods -n swiggy-clone-staging
## All pods should show STATUS: Running, RESTARTS: 0
 
## Verify the API is responding
curl -s http://swiggy-clone-staging.your-cluster.example/health/ready
## Should return: {"status": "ok"}
 
echo "✅ System healthy — ready to inject failure"

Inject

A CrashLoopBackOff happens when a pod starts, crashes immediately, and Kubernetes keeps restarting it in a loop. This usually means the application cannot start — often because a required environment variable or dependency is missing.

Bash

## Inject a bad environment variable — clear the database host
## The backend expects DB_HOST to be set — clearing it causes it to crash on startup
kubectl set env deployment/backend DB_HOST="" -n swiggy-clone-staging
 
## Watch what happens
kubectl get pods -n swiggy-clone-staging --watch
## Within 30 seconds you will see:
## NAME                       READY   STATUS             RESTARTS
## backend-xxx                0/1     CrashLoopBackOff   3
## backend-xxx                0/1     Error              3
## The restart count keeps climbing

Detect

Bash

## Step 1: Confirm the CrashLoopBackOff
kubectl get pods -n swiggy-clone-staging
## STATUS column shows CrashLoopBackOff — the pod is crashing repeatedly
 
## Step 2: Read the current crash logs
kubectl logs deployment/backend -n swiggy-clone-staging
## May be empty if the crash is very fast — use --previous instead
 
## Step 3: Read logs from the PREVIOUS container run
## --previous reads logs from the last container that crashed
kubectl logs deployment/backend -n swiggy-clone-staging --previous
## Look for: "DB_HOST is required", "Cannot connect to database",
## "Connection refused", or any startup error from your application code
 
## Step 4: Describe the pod for more detail
kubectl describe pod -l app=backend -n swiggy-clone-staging | tail -30
## Look for the Events section:
## Warning  BackOff  2m   kubelet  Back-off restarting failed container
 
## Step 5: Check if it is an environment variable problem
kubectl exec deployment/backend -n swiggy-clone-staging -- env | grep DB
## If the pod can start long enough to exec, you will see DB_HOST is empty
## If it cannot, this command fails — which itself confirms the startup crash

Remember
kubectl logs without --previous shows the current container's logs. If a pod crashes immediately, there are no current logs. Always try --previous first for CrashLoopBackOff.

Fix

Bash

## Restore the correct database host
kubectl set env deployment/backend \
  DB_HOST=postgres-service \
  -n swiggy-clone-staging
 
## Watch the pods recover
kubectl rollout status deployment/backend -n swiggy-clone-staging
## Output: deployment "backend" successfully rolled out
 
## Verify healthy
kubectl get pods -n swiggy-clone-staging
## STATUS should return to Running, RESTARTS will show the crash count
## but no new restarts should be occurring

Prevent

The application should validate all required environment variables at startup and crash immediately with a clear message if any are missing. This makes CrashLoopBackOff self-diagnosing — the log message tells you exactly what is wrong.

JAVASCRIPT

// src/startup-validation.js
// Add this as the first thing your application runs
 
const REQUIRED_ENV_VARS = [
  'DB_HOST',
  'DB_PORT',
  'DB_NAME',
  'DB_USER',
  'DB_PASSWORD',
  'REDIS_HOST',
  'PORT',
];
 
function validateEnvironment() {
  const missing = REQUIRED_ENV_VARS.filter(key => !process.env[key]);
 
  if (missing.length > 0) {
    // This message appears in kubectl logs --previous
    // Making it specific means the engineer reading it knows exactly what to fix
    console.error(`FATAL: Missing required environment variables: ${missing.join(', ')}`);
    console.error('The application cannot start without these values.');
    console.error('Check your Kubernetes Secret and Deployment environment config.');
    process.exit(1);   // non-zero exit causes Kubernetes to count it as a crash
  }
}
 
validateEnvironment();   // call this before any other application code

Scenario 2 — OOMKill — Pod Killed for Using Too Much Memory

What OOMKill is

When a Kubernetes pod exceeds its memory limit, the Linux kernel's OOM (Out of Memory) killer terminates the process. Kubernetes then restarts it. Unlike a crash from a bug, the pod did not fail — it was killed by the system for consuming too many resources.

OOMKill is common in two situations: the memory limit is set too low for the actual workload, or the application has a memory leak that causes usage to grow over time until it hits the ceiling.

Setup

Bash

## Set a very low memory limit — 32Mi is too small for a Node.js application
kubectl set resources deployment/backend \
  -c backend \
  --limits=memory=32Mi \
  -n swiggy-clone-staging
 
## Wait for pods to restart with new limits
kubectl rollout status deployment/backend -n swiggy-clone-staging

Inject

Bash

## Send requests that cause memory usage to spike
## Install 'hey' load testing tool if not installed
go install github.com/rakyll/hey@latest
 
## Send 500 concurrent requests — this drives memory usage up
hey -n 1000 -c 50 http://swiggy-clone-staging.your-cluster.example/api/menu
## You do not need to wait for this to finish — watch the pods in another terminal

Detect

Bash

## Terminal 1: Watch pods in real time
kubectl get pods -n swiggy-clone-staging --watch
## You will see the pod status change:
## backend-xxx   1/1   Running   0   → 0/1   OOMKilled   1
 
## Terminal 2: Describe the pod after it crashes
kubectl describe pod -l app=backend -n swiggy-clone-staging | grep -A 10 "Last State"
## Output:
## Last State: Terminated
##   Reason: OOMKilled        ← this is the key indicator
##   Exit Code: 137           ← exit code 137 = killed by signal 9 (SIGKILL)
 
## Check Prometheus for memory usage
## Run this PromQL query in Grafana or the Prometheus UI:
## container_memory_working_set_bytes{namespace="swiggy-clone-staging", container="backend"}
## You will see the spike just before the OOMKill
 
## Check Grafana for the memory spike
## Dashboard: Kubernetes / Compute Resources / Pod
## Filter: namespace=swiggy-clone-staging, pod=backend-xxx
## The memory graph will show a sharp climb to 32Mi then drop to zero (restart)

Remember
Exit code 137 always means OOMKill. Exit code 1 means the application itself exited with an error. Exit code 0 means the application exited cleanly (probably a job that finished). These codes are your first diagnostic signal.

Fix

Bash

## Step 1: Find out how much memory the backend actually uses under normal load
## Query Prometheus for P95 memory usage over the last 24 hours
## PromQL: quantile_over_time(0.95,
##   container_memory_working_set_bytes{
##     namespace="swiggy-clone-staging",
##     container="backend"
##   }[24h])
## If result is ~180Mi, set limit to 256Mi (1.5x headroom is a good rule)
 
## Step 2: Set a sensible memory limit based on actual usage
kubectl set resources deployment/backend \
  -c backend \
  --requests=memory=128Mi \
  --limits=memory=512Mi \
  -n swiggy-clone-staging
 
## Step 3: Verify pods restart cleanly
kubectl rollout status deployment/backend -n swiggy-clone-staging
kubectl get pods -n swiggy-clone-staging
## RESTARTS should not be climbing anymore

Prevent

Bash

## Install Vertical Pod Autoscaler (VPA) from the FinOps module
## VPA watches actual CPU/memory usage and recommends correct limits
 
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vertical-pod-autoscaler.yaml
 
## Create a VPA object for the backend in recommendation-only mode
## Recommendation mode: suggests values but does not change them automatically
## This is the safest starting point — you review recommendations before applying

YAML

## k8s/vpa-backend.yaml
 
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: backend-vpa
  namespace: swiggy-clone-staging
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend
  updatePolicy:
    updateMode: "Off"   ## Off = recommend only, do not change pods automatically

Bash

kubectl apply -f k8s/vpa-backend.yaml
 
## After 24 hours, check the VPA recommendations
kubectl describe vpa backend-vpa -n swiggy-clone-staging | grep -A 10 Recommendation
## Output:
## Lower Bound: cpu: 50m, memory: 100Mi
## Target: cpu: 120m, memory: 180Mi     ← set your limits based on Target
## Upper Bound: cpu: 300m, memory: 400Mi
## Uncapped Target: cpu: 120m, memory: 180Mi

Common Mistake
Sizing memory limits based on average usage. If average is 100Mi but P95 is 180Mi, setting a limit of 120Mi will OOMKill during traffic spikes. Always size limits from the P95 or P99 percentile, not the average.

Scenario 3 — Bad Deployment — Rollout Goes Wrong

Setup

Bash

## Verify a clean starting state
kubectl get rollout backend -n swiggy-clone-staging
## STATUS: Healthy, STABLE: backend:v1.5.0 or similar
 
## Confirm the application is responding
curl -s http://swiggy-clone-staging.your-cluster.example/health/ready
## Returns: {"status": "ok"}

Inject

Bash

## Deploy an image tag that does not exist in the registry
## This simulates a CI pipeline that accidentally pushed a wrong tag name
kubectl set image rollout/backend \
  backend=your-account.dkr.ecr.ap-south-1.amazonaws.com/swiggy-backend:nonexistent-broken \
  -n swiggy-clone-staging
 
## Watch what happens
kubectl get pods -n swiggy-clone-staging --watch
## STATUS changes to: ImagePullBackOff
## The rollout is stuck — new pods cannot start, old pods are still running

Detect

Bash

## Check the pod status
kubectl get pods -n swiggy-clone-staging
## backend-new-xxx   0/1   ImagePullBackOff   0
 
## Describe the failing pod
kubectl describe pod -l app=backend -n swiggy-clone-staging | grep -A 5 "Events:"
## Events:
##   Warning  Failed  30s  kubelet  Failed to pull image: not found
 
## Check ArgoCD — it shows the rollout health
argocd app get swiggy-clone-staging
## Health Status: Degraded
## This tells you ArgoCD detected the rollout did not complete successfully
 
## Check the Argo Rollout status directly
kubectl argo rollouts get rollout backend -n swiggy-clone-staging
## Status:    ✖ Degraded
## Strategy:  Canary
## Step: 0/7
## The canary step shows the broken image tag — the old stable version
## is still serving 100% of traffic because the canary never passed Step 1

Fix

Bash

## Option 1: ArgoCD rollback via the UI
## Open the ArgoCD UI → swiggy-clone-staging → HISTORY AND ROLLBACK
## Click Rollback on the previous healthy revision → Confirm
 
## Option 2: kubectl rollout undo (fast, terminal-only)
kubectl argo rollouts undo rollout/backend -n swiggy-clone-staging
## This immediately restores the previous stable image tag
 
## Option 3: Fix forward — deploy the correct image tag
kubectl set image rollout/backend \
  backend=your-account.dkr.ecr.ap-south-1.amazonaws.com/swiggy-backend:v1.5.0 \
  -n swiggy-clone-staging
 
## Verify recovery
kubectl argo rollouts get rollout backend -n swiggy-clone-staging
## Status: Healthy

Prevent

Add image tag validation to the CI pipeline so a tag that does not exist in the registry cannot be deployed.

YAML

## .github/workflows/deploy.yml
## Add this validation step before updating the Kubernetes manifest
 
- name: Validate image exists in ECR before deploying
  run: |
    ## Check if the image tag exists in ECR
    ## This fails the pipeline if the image was not pushed successfully
    aws ecr describe-images \
      --repository-name swiggy-backend \
      --image-ids imageTag=${IMAGE_TAG} \
      --region ap-south-1
    echo "✅ Image ${IMAGE_TAG} confirmed in ECR — safe to deploy"

Scenario 4 — Database Connection Exhaustion

What connection exhaustion is

Every PostgreSQL database has a maximum number of simultaneous connections (max_connections). The default is 100. If you have 20 pods each opening 10 connections, you have 200 connections — exceeding the limit. New connection attempts fail with FATAL: remaining connection slots are reserved. Every API request that needs the database returns a 500 error.

Without a connection pooler like PgBouncer, every pod holds open database connections for its entire lifetime. As you scale out, connection count grows linearly. With PgBouncer, pods connect to PgBouncer (which can handle thousands of connections) and PgBouncer maintains a smaller pool to the actual database.

Setup

Bash

## Check current connection count — baseline before injection
kubectl exec -n swiggy-clone-staging deployment/postgres -- \
  psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
## Note this number — likely 5-10 with a small deployment

Inject

Bash

## Remove connection pool limits from the backend and scale to 20 replicas
## Each replica will open a new pool of connections to PostgreSQL
kubectl set env deployment/backend \
  DB_POOL_SIZE=20 \
  -n swiggy-clone-staging
 
kubectl scale deployment backend --replicas=20 -n swiggy-clone-staging
 
## Watch connection count climb
kubectl exec -n swiggy-clone-staging deployment/postgres -- \
  psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
## Runs the query every 5 seconds — watch the count climb toward 100
 
## Send requests to trigger connection failures
hey -n 500 -c 100 http://swiggy-clone-staging.your-cluster.example/api/menu
## API will start returning 500 errors when connection limit is hit

Detect

Bash

## Check PostgreSQL logs for connection errors
kubectl logs deployment/postgres -n swiggy-clone-staging | \
  grep -i "connection\|too many\|remaining"
## Look for: FATAL: remaining connection slots are reserved for non-replication
## Look for: FATAL: sorry, too many clients already
 
## Check the API error rate in Prometheus
## PromQL: rate(http_requests_total{status=~"5..",app="backend"}[5m])
## You will see a spike in 5xx errors coinciding with scale-out
 
## Check connection count by application
kubectl exec -n swiggy-clone-staging deployment/postgres -- \
  psql -U postgres -c "
    SELECT application_name, count(*)
    FROM pg_stat_activity
    GROUP BY application_name
    ORDER BY count DESC;"
## This shows which application is consuming connections

Fix

Bash

## Step 1: Immediately scale back down to stop the bleeding
kubectl scale deployment backend --replicas=3 -n swiggy-clone-staging
 
## Step 2: Restore sensible connection pool size
kubectl set env deployment/backend \
  DB_POOL_SIZE=5 \
  -n swiggy-clone-staging
 
## Step 3: Deploy PgBouncer as a connection pooler

YAML

## k8s/pgbouncer.yaml
## PgBouncer sits between the application and PostgreSQL
## Pods connect to PgBouncer instead of PostgreSQL directly
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbouncer
  namespace: swiggy-clone-staging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pgbouncer
  template:
    metadata:
      labels:
        app: pgbouncer
    spec:
      containers:
      - name: pgbouncer
        image: pgbouncer/pgbouncer:1.21.0
        ports:
        - containerPort: 5432
        env:
        - name: DATABASES_HOST
          value: postgres-service   ## the real PostgreSQL service name
        - name: DATABASES_PORT
          value: "5432"
        - name: DATABASES_DBNAME
          value: swiggy
        - name: PGBOUNCER_POOL_MODE
          value: transaction   ## transaction pooling: most efficient
        - name: PGBOUNCER_MAX_CLIENT_CONN
          value: "1000"        ## PgBouncer accepts up to 1000 connections
        - name: PGBOUNCER_DEFAULT_POOL_SIZE
          value: "20"          ## PgBouncer keeps only 20 connections to PostgreSQL
        resources:
          requests:
            cpu: "50m"
            memory: "32Mi"
          limits:
            cpu: "200m"
            memory: "64Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: pgbouncer-service
  namespace: swiggy-clone-staging
spec:
  selector:
    app: pgbouncer
  ports:
  - port: 5432
    targetPort: 5432

Bash

kubectl apply -f k8s/pgbouncer.yaml
 
## Update the backend to connect to PgBouncer instead of PostgreSQL directly
kubectl set env deployment/backend \
  DB_HOST=pgbouncer-service \
  -n swiggy-clone-staging
 
## Verify connection count dropped
kubectl exec -n swiggy-clone-staging deployment/postgres -- \
  psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
## Should now show ~20 connections (PgBouncer pool) regardless of backend replica count

Prevent

YAML

## k8s/resource-quota.yaml
## ResourceQuota limits the maximum number of replicas in a namespace
## This prevents accidental scale-out that exhausts database connections
 
apiVersion: v1
kind: ResourceQuota
metadata:
  name: namespace-limits
  namespace: swiggy-clone-staging
spec:
  hard:
    ## Maximum 10 pods in this namespace — prevents runaway scaling
    pods: "10"
    ## Maximum CPU and memory across all pods in the namespace
    requests.cpu: "4"
    requests.memory: 4Gi
    limits.cpu: "8"
    limits.memory: 8Gi

Scenario 5 — Cost Spike Investigation

Setup

Bash

## Verify Kubecost is installed (from the FinOps module)
kubectl get pods -n kubecost
## Should show kubecost-cost-analyzer Running
 
## Open the Kubecost dashboard and note the current daily cost
## http://localhost:9090 (after port-forward)
kubectl port-forward -n kubecost deployment/kubecost-cost-analyzer 9090:9090 &
## Note the current cost for swiggy-clone-staging namespace as a baseline

Inject

Bash

## Create a Deployment with no resource requests and scale it to 50 replicas
## Without resource requests, Kubernetes cannot schedule efficiently
## Pods claim no resources on paper but consume actual CPU and memory
## Kubecost cannot correctly attribute cost — and the waste is enormous
 
kubectl create deployment cost-hog \
  --image=nginx:latest \
  -n swiggy-clone-staging
 
## Scale to 50 replicas immediately
kubectl scale deployment cost-hog --replicas=50 -n swiggy-clone-staging
 
## Watch nodes become pressured
kubectl describe nodes | grep -A 5 "Allocated resources"
## CPU and memory allocation will spike significantly

Detect

Bash

## Open Kubecost dashboard at http://localhost:9090
## Navigate to: Allocations → Namespace → swiggy-clone-staging
## You will see a sudden spike in cost for the cost-hog deployment
## The efficiency score will drop because there are no resource requests set
 
## Check from the terminal
kubectl get deployment cost-hog -n swiggy-clone-staging
## READY column will show 50/50 — 50 pods consuming untracked resources
 
## Check cluster capacity impact
kubectl top nodes
## CPU% and Memory% on nodes will have jumped significantly
 
## Prometheus alert would have fired (if configured from FinOps module):
## KubecostBudgetExceeded: namespace swiggy-clone-staging exceeded daily budget

Fix

Bash

## Delete the runaway Deployment immediately
kubectl delete deployment cost-hog -n swiggy-clone-staging
 
## Verify cost returns to baseline
kubectl top nodes
## CPU% and Memory% should return to previous levels within 2 minutes
 
## Check Kubecost shows the cost dropping back to baseline
## This may take up to 15 minutes for Kubecost to reflect the change

Prevent

YAML

## k8s/limit-range.yaml
## LimitRange sets default resource requests on pods that do not specify them
## It also sets a minimum — pods cannot be created without at least these resources
 
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: swiggy-clone-staging
spec:
  limits:
  - type: Container
    ## If a pod does not specify requests, these defaults are applied automatically
    defaultRequest:
      cpu: "50m"
      memory: "64Mi"
    ## If a pod does not specify limits, these defaults are applied
    default:
      cpu: "500m"
      memory: "512Mi"
    ## Minimum allowed values — a pod cannot request less than these
    min:
      cpu: "10m"
      memory: "32Mi"
    ## Maximum allowed values — a pod cannot request more than these
    max:
      cpu: "2"
      memory: "4Gi"

YAML

## k8s/kyverno-require-resources.yaml
## Kyverno policy: blocks any Deployment that does not have resource requests
## This is enforced at admission time — the Deployment cannot be created
 
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-requests
spec:
  validationFailureAction: enforce   ## block, do not just warn
  rules:
  - name: require-requests
    match:
      any:
      - resources:
          kinds:
          - Deployment
    validate:
      message: "Resource requests (CPU and memory) are required on all containers. Set spec.containers[].resources.requests."
      pattern:
        spec:
          template:
            spec:
              containers:
              - resources:
                  requests:
                    memory: "?*"   ## ?* means: must be present and non-empty
                    cpu: "?*"

Scenario 6 — Network Policy Lockdown

Setup

Bash

## Verify pods can communicate normally
kubectl exec -n swiggy-clone-staging deployment/backend -- \
  curl -s http://postgres-service:5432
## Should respond (even if it is a connection refused — that means networking works)
 
kubectl exec -n swiggy-clone-staging deployment/backend -- \
  nslookup google.com
## DNS should resolve successfully

Inject

Bash

## Apply an overly restrictive NetworkPolicy that blocks all traffic
## including DNS resolution — a very common mistake
 
kubectl apply -f - << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-broken
  namespace: swiggy-clone-staging
spec:
  podSelector: {}   ## applies to ALL pods in this namespace
  policyTypes:
  - Ingress
  - Egress
  ## No ingress or egress rules = deny everything
  ## This blocks DNS (port 53), database connections, everything
EOF
 
## Watch pods start failing readiness probes
kubectl get pods -n swiggy-clone-staging --watch
## READY changes from 1/1 to 0/1 within 30 seconds
## Readiness probes fail because backend cannot reach PostgreSQL

Detect

Bash

## Test DNS resolution from inside a pod
kubectl exec -n swiggy-clone-staging deployment/backend -- \
  nslookup postgres-service
## Error: connection timed out; no servers could be reached
## This confirms DNS is blocked — the NetworkPolicy is too strict
 
## Test service connectivity
kubectl exec -n swiggy-clone-staging deployment/backend -- \
  curl -v http://postgres-service:5432 --connect-timeout 5
## Error: Connection timed out
## Confirming egress is blocked
 
## Check NetworkPolicy exists
kubectl get networkpolicy -n swiggy-clone-staging
## Shows: deny-all-broken
 
## Check which pods the policy applies to
kubectl describe networkpolicy deny-all-broken -n swiggy-clone-staging
## Pod Selector: <none> (all pods)
## Policy Types: Ingress, Egress
## No rules listed = total lockdown
 
## Check Ingress returns 503
curl -v http://swiggy-clone-staging.your-cluster.example/api/menu
## HTTP/1.1 503 Service Unavailable
## Readiness probes are failing so pods are removed from the Service's Endpoints

Fix

Bash

## Step 1: Delete the broken policy immediately
kubectl delete networkpolicy deny-all-broken -n swiggy-clone-staging
 
## Step 2: Verify recovery
kubectl get pods -n swiggy-clone-staging
## READY should return to 1/1 within 10 seconds as readiness probes pass
 
## Step 3: Apply a CORRECT NetworkPolicy that allows DNS and same-namespace traffic

YAML

## k8s/network-policy-correct.yaml
## A NetworkPolicy that follows the principle of least privilege
## but does NOT break DNS or legitimate service-to-service communication
 
## Policy 1: Allow DNS egress from all pods
## Without this, pods cannot resolve service names — everything breaks
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: swiggy-clone-staging
spec:
  podSelector: {}   ## apply to all pods
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: UDP   ## DNS uses UDP on port 53
    - port: 53
      protocol: TCP   ## some DNS queries use TCP (large responses)
---
## Policy 2: Allow traffic within the same namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: swiggy-clone-staging
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector: {}   ## allow from any pod in the same namespace
  egress:
  - to:
    - podSelector: {}   ## allow to any pod in the same namespace
---
## Policy 3: Allow ingress from the ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
  namespace: swiggy-clone-staging
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx   ## your ingress namespace

Bash

kubectl apply -f k8s/network-policy-correct.yaml
 
## Verify DNS works again
kubectl exec -n swiggy-clone-staging deployment/backend -- \
  nslookup postgres-service
## Should resolve to the ClusterIP
 
## Verify service connectivity
curl -s http://swiggy-clone-staging.your-cluster.example/api/menu
## Should return 200

Prevent

YAML

## k8s/kyverno-generate-network-policy.yaml
## Kyverno auto-generates the correct default NetworkPolicies
## for every new namespace — so they start with sensible defaults
 
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-default-network-policies
spec:
  rules:
  - name: add-allow-dns
    match:
      any:
      - resources:
          kinds:
          - Namespace
    ## generate: creates the NetworkPolicy when a new namespace is created
    generate:
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      name: allow-dns-egress
      namespace: "{{request.object.metadata.name}}"
      synchronize: true   ## keeps the policy in sync if manually deleted
      data:
        spec:
          podSelector: {}
          policyTypes:
          - Egress
          egress:
          - ports:
            - port: 53
              protocol: UDP
            - port: 53
              protocol: TCP

SLO Definition and Burn Rate Alerts

What an SLO is

An SLO (Service Level Objective) is a target reliability level for a service. It answers the question: how reliable does this service need to be for users to be satisfied?

An SLO is not a promise to users (that is an SLA). It is an internal target used to make engineering decisions. If the service is above the SLO, the team can invest in features. If it is below, the team stops feature work and fixes reliability.

For the swiggy-clone application, here are three meaningful SLOs:

SLO	Target	Meaning
Availability	99.9%	No more than 43.8 minutes of downtime per month
P95 Latency	< 500ms	95% of requests served in under half a second
Error Rate	< 1%	No more than 1 in 100 requests returns a 5xx response

Error budget

If the availability SLO is 99.9%, the error budget is 0.1% of total uptime. In a 30-day month with 43,200 minutes, the error budget is 43.2 minutes. Once that budget is exhausted, any further downtime is violating the SLO.

The error budget is the most useful concept from SRE practice. It converts an abstract percentage into concrete time that engineers can reason about. "We have 43 minutes of error budget left this month" is actionable. "We are at 99.91% availability" is not.

Prometheus recording rules and alerting rules

YAML

## monitoring/slo-rules.yaml
## PrometheusRule defines recording rules and alerting rules for SLOs
 
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: swiggy-clone-slos
  namespace: monitoring
  labels:
    release: monitoring   ## must match your Prometheus operator label selector
spec:
  groups:
  ## ─── Availability SLO ──────────────────────────────────────────
  - name: swiggy-clone-availability
    interval: 30s
    rules:
    ## Recording rule: calculate availability over a 5-minute window
    ## This pre-computes the value so alerts evaluate faster
    - record: job:http_availability:rate5m
      expr: |
        sum(
          rate(
            http_requests_total{
              namespace="swiggy-clone-production",
              app="backend",
              status!~"5.."   ## exclude 5xx errors from the success count
            }[5m]
          )
        )
        /
        sum(
          rate(
            http_requests_total{
              namespace="swiggy-clone-production",
              app="backend"
            }[5m]
          )
        )
    ## Alert: availability dropped below 99.9% (immediate — high burn rate)
    - alert: SLOAvailabilityBurnRateFast
      expr: |
        (
          1 - job:http_availability:rate5m
        ) > (14.4 * 0.001)
        ## 14.4x burn rate: consuming 1-hour budget in 5 minutes
        ## 0.001 = the 0.1% error budget
      for: 2m
      labels:
        severity: critical
        team: backend-team
      annotations:
        summary: "Availability SLO burn rate critical — action required immediately"
        description: "Swiggy-clone backend is burning error budget at 14.4x the sustainable rate. At this rate the monthly budget will be exhausted in under 1 hour."
        runbook: "https://backstage.internal/docs/default/component/swiggy-clone-backend/runbooks/availability"
 
    ## Alert: slower burn rate — will exhaust budget in ~3 days
    - alert: SLOAvailabilityBurnRateSlow
      expr: |
        (
          1 - job:http_availability:rate5m
        ) > (6 * 0.001)
        ## 6x burn rate: consuming 6-hour budget in 1 hour
      for: 15m
      labels:
        severity: warning
        team: backend-team
      annotations:
        summary: "Availability SLO burn rate elevated"
        description: "Budget will be exhausted in approximately 3 days at the current rate. Investigate root cause before it becomes critical."
 
  ## ─── Latency SLO ───────────────────────────────────────────────
  - name: swiggy-clone-latency
    rules:
    - record: job:http_request_duration_p95:rate5m
      expr: |
        histogram_quantile(
          0.95,
          sum(
            rate(
              http_request_duration_seconds_bucket{
                namespace="swiggy-clone-production",
                app="backend"
              }[5m]
            )
          ) by (le)
        )
    - alert: SLOLatencyP95Exceeded
      expr: job:http_request_duration_p95:rate5m > 0.5   ## > 500ms
      for: 5m
      labels:
        severity: warning
        team: backend-team
      annotations:
        summary: "P95 latency SLO breached"
        description: "P95 latency is {{ $value | humanizeDuration }} — SLO target is 500ms."

Remember
Burn rate alerts are more useful than threshold alerts. "The error rate is 2%" tells you the current state. "At this rate you will exhaust your monthly error budget in 4 hours" tells you how urgent the problem is and drives the right response.

Production Checklist

Bash

## ─── 1. Run all 6 scenarios and verify recovery ──────────────────
## For each scenario, verify these three things after recovery:
kubectl get pods -n swiggy-clone-staging
## All pods: STATUS=Running, RESTARTS should not be climbing
 
curl -s http://swiggy-clone-staging.your-cluster.example/health/ready
## Returns: {"status": "ok"}
 
kubectl argo rollouts get rollout backend -n swiggy-clone-staging 2>/dev/null || \
kubectl get deployment backend -n swiggy-clone-staging
## Healthy or AVAILABLE=1
 
## ─── 2. Verify startup validation is active ──────────────────────
## Intentionally break DB_HOST and check the log message is clear
kubectl set env deployment/backend DB_HOST="" -n swiggy-clone-staging
sleep 15
kubectl logs deployment/backend -n swiggy-clone-staging --previous 2>/dev/null | \
  grep -i "FATAL\|required\|missing"
## Should see the specific missing variable message from startup-validation.js
## Restore immediately:
kubectl set env deployment/backend DB_HOST=postgres-service -n swiggy-clone-staging
 
## ─── 3. Verify PgBouncer is deployed ─────────────────────────────
kubectl get deployment pgbouncer -n swiggy-clone-staging
## AVAILABLE should be 1
 
## ─── 4. Verify SLO alerting rules are loaded ─────────────────────
kubectl get prometheusrule swiggy-clone-slos -n monitoring
## Should exist
 
## Query the recording rule to verify it is computing
curl -s "http://localhost:9090/api/v1/query?query=job:http_availability:rate5m" | \
  jq '.data.result[0].value[1]'
## Should return a number between 0.99 and 1.0
 
## ─── 5. Verify NetworkPolicies are correct ───────────────────────
kubectl get networkpolicy -n swiggy-clone-staging
## Should show: allow-dns-egress, allow-same-namespace, allow-ingress-controller
## Should NOT show: deny-all-broken
 
## Verify DNS still works from inside the namespace
kubectl run dns-test --image=busybox --restart=Never -n swiggy-clone-staging \
  --rm -it -- nslookup postgres-service
## Should resolve without errors
## Pod is automatically deleted after the test (--rm flag)
 
## ─── 6. Verify LimitRange is enforced ───────────────────────────
kubectl get limitrange default-limits -n swiggy-clone-staging
## Should exist
 
## Test: try to create a pod without resource requests
kubectl run no-resources --image=nginx -n swiggy-clone-staging --restart=Never
kubectl get pod no-resources -n swiggy-clone-staging -o json | \
  jq '.spec.containers[0].resources'
## Should show default requests/limits applied by LimitRange
kubectl delete pod no-resources -n swiggy-clone-staging
 
echo "✅ Production readiness checklist complete"

Common Production Mistakes

❌ Running chaos experiments in production before staging 💥 An engineer decides to test how the production cluster handles a node failure. They drain a node during peak lunch hours. Three services were only running one replica each. They all go down simultaneously. The ordering system is unavailable for 12 minutes. Millions of rupees in orders are lost. ✅ Every chaos experiment must run in staging first. Only when you have validated recovery in staging, documented the playbook, scheduled a maintenance window, and have the full on-call team available should you consider running it in production. Netflix did not start Chaos Monkey in production on day one — they built it in a non-production environment, validated it for months, then gradually expanded.

❌ No runbook for each alert 💥 An SLOAvailabilityBurnRateFast alert fires at 3:47 AM. The on-call engineer opens PagerDuty, sees the alert, opens Grafana, and stares at it. The alert message says the error rate is high. The engineer does not know which service, which endpoint, or what the previous incidents looked like. They spend 20 minutes just figuring out where to start. The error budget burns while they investigate. ✅ Every alert must have a runbook annotation with a URL to a step-by-step guide. The guide must answer: what does this alert mean, what are the first three commands to run, what are the likely causes, and how have we fixed it before? The TechDocs system from Capstone 4 is the right place to store runbooks — searchable, versioned, discoverable from Backstage.

❌ Setting SLOs without measuring the baseline first 💥 A team defines an availability SLO of 99.9% for a new service on launch day. Three months later, they look at the data and discover the service has never been above 99.5%. The SLO was aspirational, not grounded in reality. Now every month they are in SLO breach, error budgets are permanently exhausted, and the engineering team is demoralized because they cannot ship features. ✅ Measure before you commit. Run the service for two to four weeks, collect availability and latency data, and set the SLO at the P10 or P25 of historical performance — better than the worst days, achievable without heroic effort. A 99.5% SLO you reliably hit is more valuable than a 99.9% SLO you constantly breach.

❌ Fixing the symptom instead of the cause 💥 The backend is OOMKilled repeatedly. The on-call engineer increases the memory limit from 256Mi to 512Mi. The OOMKills stop. The engineer closes the incident. Three weeks later, memory limit reaches 512Mi and OOMKills start again. This happens three more times until the limit is 4Gi — 16x the original value. Nobody investigated why memory keeps growing. The service has a memory leak that has been running unchecked for months. ✅ Increasing the memory limit is a valid immediate mitigation — it stops the bleeding. But it must be followed by a root cause investigation. Use a memory profiler (Node.js clinic heap or --inspect with Chrome DevTools) to find what is allocating memory. A growing heap that never shrinks is almost always a leak: event listeners not removed, caches without eviction policies, circular references in long-lived objects.

Debugging Playbook

The production debugging flowchart — the exact commands for every common failure.

Pod not running

Bash

## Step 1: What state is the pod in?
kubectl get pods -n YOUR_NAMESPACE
## CrashLoopBackOff → go to Section A
## ImagePullBackOff → go to Section B
## Pending → go to Section C
## OOMKilled → go to Section D
## Completed → go to Section E
 
## ── Section A: CrashLoopBackOff ────────────────────────────────
kubectl logs POD_NAME -n YOUR_NAMESPACE --previous
## Read the last 20 lines — look for: FATAL, Error, Cannot connect, Missing
 
kubectl describe pod POD_NAME -n YOUR_NAMESPACE | grep -A 10 "Last State"
## Check Exit Code: 1 = app error, 137 = OOMKill, 139 = segfault
 
## If logs are empty (crash is too fast):
kubectl get events -n YOUR_NAMESPACE --sort-by='.lastTimestamp' | tail -20
## Events show what Kubernetes observed
 
## ── Section B: ImagePullBackOff ─────────────────────────────────
kubectl describe pod POD_NAME -n YOUR_NAMESPACE | grep -A 5 Events
## Look for: ErrImagePull, ImagePullBackOff, unauthorized, not found
 
## Check if the image exists in the registry:
aws ecr describe-images --repository-name YOUR_REPO \
  --image-ids imageTag=YOUR_TAG --region ap-south-1
## If this fails: the image tag does not exist — roll back the deployment
 
## Check if the ECR credentials are valid in the cluster:
kubectl get secret regcred -n YOUR_NAMESPACE
## If missing: the image pull secret is not configured
 
## ── Section C: Pending ──────────────────────────────────────────
kubectl describe pod POD_NAME -n YOUR_NAMESPACE | grep -A 10 Events
## Look for: Insufficient cpu, Insufficient memory, Unschedulable
 
## Check node capacity:
kubectl describe nodes | grep -A 5 "Allocated resources"
## If nodes are at 95%+ CPU or memory: cluster needs scaling
 
## Check for taints or node selectors:
kubectl describe pod POD_NAME -n YOUR_NAMESPACE | grep -A 5 "Node-Selectors"
## If nodeSelector is set, verify nodes with that label exist:
kubectl get nodes -l YOUR_LABEL=YOUR_VALUE
 
## ── Section D: OOMKilled ────────────────────────────────────────
kubectl describe pod POD_NAME -n YOUR_NAMESPACE | grep -A 5 "Last State"
## Reason: OOMKilled, Exit Code: 137
 
## Find actual memory usage before the kill:
kubectl top pod POD_NAME -n YOUR_NAMESPACE
## Compare to the memory limit in the Deployment
 
## Check VPA recommendation if installed:
kubectl describe vpa POD_NAME-vpa -n YOUR_NAMESPACE | grep -A 10 Recommendation
 
## ── Section E: Completed ────────────────────────────────────────
## Status Completed is normal for Jobs and CronJobs
## For Deployments: Completed means the container exited with code 0
## This usually means the CMD in the Dockerfile exited — not a crash
 
kubectl logs POD_NAME -n YOUR_NAMESPACE
## Should show the final output before exit
## Ensure your Dockerfile CMD keeps the process running (not a one-shot script)

Service returning errors

Bash

## Step 1: Determine the error type
curl -v http://YOUR_SERVICE_URL/health/ready
## 502 Bad Gateway → Section F
## 503 Service Unavailable → Section G
## 500 Internal Server Error → Section H
## Timeout (no response) → Section I
 
## ── Section F: 502 Bad Gateway ──────────────────────────────────
## 502 means the Ingress reached the pod but the pod returned a bad response
## or the connection to the pod was refused
 
kubectl get endpoints YOUR_SERVICE_NAME -n YOUR_NAMESPACE
## Verify endpoints are populated — if empty, pods are not ready
 
kubectl get pods -n YOUR_NAMESPACE -l app=YOUR_APP
## Check READY column — 0/1 means readiness probe is failing
 
kubectl logs -l app=YOUR_APP -n YOUR_NAMESPACE | tail -30
## Look for application-level errors
 
## ── Section G: 503 Service Unavailable ──────────────────────────
## 503 means no ready pods exist to handle the request
 
kubectl get endpoints YOUR_SERVICE_NAME -n YOUR_NAMESPACE
## Output: (none) or empty list — no pods are ready
 
kubectl get pods -n YOUR_NAMESPACE
## Likely CrashLoopBackOff or all pods restarting simultaneously
 
## Check if NetworkPolicy is blocking Ingress:
kubectl get networkpolicy -n YOUR_NAMESPACE
## If deny-all exists with no Ingress rules: add allow-ingress-controller policy
 
## ── Section H: 500 Internal Server Error ────────────────────────
## 500 means the application is running but returning errors
 
kubectl logs -l app=YOUR_APP -n YOUR_NAMESPACE | grep -i "error\|exception" | tail -20
## Find the specific error — database connection? missing config? code bug?
 
## Check database connectivity from the pod:
kubectl exec deployment/YOUR_APP -n YOUR_NAMESPACE -- \
  nc -zv postgres-service 5432
## If fails: database is unreachable — check NetworkPolicy, check postgres pod
 
## Check error rate in Prometheus:
## PromQL: rate(http_requests_total{status=~"5..",app="YOUR_APP"}[5m])
 
## ── Section I: Timeout ──────────────────────────────────────────
## No response at all — likely network-level block or pod not reachable
 
kubectl get pods -n YOUR_NAMESPACE -l app=YOUR_APP
## Are pods running?
 
## Check NetworkPolicy:
kubectl exec deployment/YOUR_APP -n YOUR_NAMESPACE -- \
  nslookup google.com
## If DNS fails: NetworkPolicy is blocking egress including port 53
 
## Check Ingress:
kubectl describe ingress -n YOUR_NAMESPACE
## Verify the backend service name and port match what exists

Performance degraded

Bash

## Step 1: Identify where the slowness is
## PromQL: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{app="backend"}[5m]))
## If P95 latency is high, find the bottleneck:
 
## Check CPU throttling:
## PromQL: rate(container_cpu_cfs_throttled_seconds_total{container="backend"}[5m])
## If non-zero: CPU limit is too low, requests are being throttled
 
## Check memory pressure:
kubectl top pods -n YOUR_NAMESPACE
## Compare actual memory to the limit in the Deployment
 
## Check database query time:
kubectl exec deployment/postgres -n YOUR_NAMESPACE -- \
  psql -U postgres -c "
    SELECT query, mean_exec_time, calls
    FROM pg_stat_statements
    ORDER BY mean_exec_time DESC
    LIMIT 10;"
## Shows the slowest queries — missing index? N+1 problem? Lock contention?
 
## Check cache hit rate:
kubectl exec deployment/redis -n YOUR_NAMESPACE -- \
  redis-cli INFO stats | grep -E "keyspace_hits|keyspace_misses"
## If miss rate is high: cache is cold or keys are not being set correctly

What You Have Built

Five capstones. Each one building on the last. Each one mapping directly to how production Platform Engineering teams work.

◈ DIAGRAM

✅ Capstone 1: Built a cloud-native application
     Node.js backend, React frontend, PostgreSQL, Redis
     Kubernetes manifests, health probes, HPA, PDB
 
✅ Capstone 2: Provisioned production AWS infrastructure
     EKS cluster with Terraform, VPC, RDS, ElastiCache
     Prometheus and Grafana, Kubecost, IAM with least privilege
 
✅ Capstone 3: Built a GitOps delivery platform
     ArgoCD App of Apps, Kustomize overlays
     Canary deployments with Argo Rollouts
     Staging → production promotion via pull request
 
✅ Capstone 4: Built a developer portal and golden paths
     Backstage software catalog with auto-discovery
     Golden Path Template: new service in 10 minutes
     TechDocs: docs as code, always current
 
✅ Capstone 5: Operated it under real failure conditions
     CrashLoopBackOff: diagnosed and prevented with startup validation
     OOMKill: root-caused and prevented with VPA
     Bad deployment: rolled back via ArgoCD in under 2 minutes
     Connection exhaustion: fixed with PgBouncer
     Cost spike: detected with Kubecost, prevented with Kyverno
     Network lockdown: debugged NetworkPolicy step by step
     SLOs defined, error budgets calculated, burn rate alerts configured

Someone who has completed all five of these capstones has not just studied Platform Engineering. They have done it.

Every scenario in Capstone 5 is real. Not invented for teaching purposes. OOMKills happen at Hotstar during IPL. Connection exhaustion happens at Razorpay during Diwali. Bad deployments happen at every company that ships software. Network policies that are too strict have broken more production environments than deliberate attacks.

Every concept, every tool, every decision maps directly to how production Platform Engineering teams work at Indian product companies.

You are ready.

Videos & Guides

No external walkthroughs, video tutorials, or reference files attached to this blueprint yet.