Master core concepts and production patterns.
Netflix famously runs Chaos Monkey in production — a program that randomly kills production services to confirm the system can survive failures. The logic: if you never deliberately break things, you never know how they behave when they break on their own — and they will, at the worst possible time. **Chaos engineering** is the discipline of deliberately injecting failure into systems in a controlled way, to discover weaknesses before they become real incidents. This project builds a complete chaos engineering practice on Kubernetes using Chaos Mesh, with k6 generating realistic load and Grafana showing the real-time impact — the same practice an SRE team at Hotstar would run before a high-traffic event like an IPL final. [ Define steady state ] (SLOs: p99 latency < 200ms, error rate < 0.5%) | v [ Form hypothesis ] ("Service maintains SLO when 1 of 3 pods is killed") | v [ Inject failure with Chaos Mesh ] (Kill a pod while k6 sends continuous traffic) | v [ Observe on Grafana ] (Did latency spike? Did errors occur? Did it recover?) | v [ Document findings ] (Hypothesis confirmed or refuted - what did we learn?) | v [ Fix the weaknesses found ] (Add PDB, increase replicas, add timeouts) > 💡 **Tip:** You will run this same five-step cycle — hypothesis, inject, observe, document, fix — three separate times against three different failure types. The cycle itself is the actual skill being practiced, not any one experiment.
Most teams only discover their system's real failure behavior during an actual incident — at 2am, under real user impact, with no controlled way to isolate what broke. By then it's too late to learn calmly; the priority is restoring service, not understanding the system. Chaos engineering solves this by moving failure discovery into a controlled, scheduled, low-stakes setting. You define a **steady state hypothesis** — a measurable statement of what "normal" looks like, backed by real metrics, not a guess. You then inject a specific failure and observe whether the steady state holds. If your hypothesis is refuted, you found a genuine weakness on your own terms, with time to fix it, instead of during a customer-facing outage during peak order volume at a company like Swiggy. > ⚠️ **Security:** Never run chaos experiments in production before validating the same experiment in staging. This project builds the practice safely on a local cluster — treat that discipline as non-negotiable before ever pointing these tools at a live environment. ---
Master core concepts and production patterns.
Chaos engineering requires something with redundancy to test against. A single-replica deployment can't teach you anything useful — killing its only pod is simply an outage, not an experiment. Three replicas is the practical minimum for a meaningful pod-kill test.
```bash kubectl create namespace chaos-demo cat > target-app.yaml << 'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: orders-api namespace: chaos-demo spec: replicas: 3 selector: matchLabels: app: orders-api template: metadata: labels: app: orders-api spec: containers: - name: api image: nginx:alpine ports: - containerPort: 80 resources: requests: memory: "64Mi" cpu: "100m" limits: memory: "128Mi" cpu: "200m" readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 3 periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: name: orders-api namespace: chaos-demo spec: selector: app: orders-api ports: - port: 80 targetPort: 80 EOF kubectl apply -f target-app.yaml kubectl get pods -n chaos-demo ``` ```text NAME READY STATUS orders-api-7d9f8c6b-2kmpq 1/1 Running orders-api-7d9f8c6b-5xnpl 1/1 Running orders-api-7d9f8c6b-9rvbf 1/1 Running ``` > 🔴 **Common Mistake:** Running pod-kill experiments against a single-replica deployment isn't a chaos experiment — it's just an outage. Chaos engineering requires redundancy to actually test against. ---
Master this concept and view production exercises.
Netflix famously runs Chaos Monkey in production — a program that randomly kills production services to confirm the syst...
Most teams only discover their system's real failure behavior during an actual incident — at 2am, under real user impact...
Master this concept and view production exercises.
Chaos engineering requires something with redundancy to test against. A single-replica deployment can't teach you anythi...
> 🔴 Common Mistake: Running pod-kill experiments against a single-replica deployment isn't a chaos experiment — it's ju...
Master this concept and view production exercises.
Chaos Mesh is a CNCF project providing Kubernetes-native chaos engineering. Instead of writing ad-hoc scripts to kill pr...
Access the dashboard: Open http://localhost:2333 in your browser. > 💡 Tip: The Chaos Mesh dashboard lets you build and ...
Master this concept and view production exercises.
k6 generates realistic HTTP traffic during the experiment so you're measuring the failure's effect on real simulated loa...
Let it run to steady state before injecting any chaos — this baseline run is what every experiment result gets compared ...
Master this concept and view production exercises.
Hypothesis: When one of three orders-api pods is killed, the remaining two pods absorb the traffic without the error rat...
Watch the k6 output during the experiment for error spikes or latency increases during the restart window. > 🔴 Common M...
Master this concept and view production exercises.
Hypothesis: When 300ms of network delay is injected on outbound traffic from the orders-api pods, users will observe lat...
Watch k6's p99 latency metric during this run — the question isn't whether it rises, but whether it stays under your SLO...
Master this concept and view production exercises.
Hypothesis: When CPU is throttled to 50% of the container limit across all pods, the service will maintain its error rat...
The experiment cleans itself up automatically after 2 minutes. > 🔴 Common Mistake: Not setting an explicit duration on ...
Master this concept and view production exercises.
Documenting results is as important as running the experiment — an experiment without a written finding gets forgotten b...
> 💡 Tip: Store chaos-report.md in the same Git repo as your Kubernetes manifests. Over time, a folder of these reports ...
Master this concept and view production exercises.
A Pod Disruption Budget tells Kubernetes the minimum number of pods that must remain available during any voluntary disr...
Re-run Experiment 1 (Milestone 4) with the PDB in place. The pod kill should now cause noticeably less disruption, since...
Master this concept and view production exercises.
> 💡 Tip: Compare the k6 threshold pass/fail output (not just eyeballing the numbers) between the first and second pod-k...
Mistake Why It Breaks Fix No observability before chaos Can't tell if the experiment did anything Install Prometheus/Gra...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.