Run a Chaos Engineering Lab with Chaos Mesh, k6, and Grafana

Netflix famously runs Chaos Monkey in production — a program that randomly kills production services to confirm the system can survive failures. The logic: if you never deliberately break things, you never know how they behave when they break on their own — and they will, at the worst possible time.

Chaos engineering is the discipline of deliberately injecting failure into systems in a controlled way, to discover weaknesses before they become real incidents. This project builds a complete chaos engineering practice on Kubernetes using Chaos Mesh, with k6 generating realistic load and Grafana showing the real-time impact — the same practice an SRE team at Hotstar would run before a high-traffic event like an IPL final.

Tip

You will run this same five-step cycle — hypothesis, inject, observe, document, fix — three separate times against three different failure types. The cycle itself is the actual skill being practiced, not any one experiment.

Most teams only discover their system's real failure behavior during an actual incident — at 2am, under real user impact, with no controlled way to isolate what broke. By then it's too late to learn calmly; the priority is restoring service, not understanding the system.

Chaos engineering solves this by moving failure discovery into a controlled, scheduled, low-stakes setting. You define a steady state hypothesis — a measurable statement of what "normal" looks like, backed by real metrics, not a guess. You then inject a specific failure and observe whether the steady state holds. If your hypothesis is refuted, you found a genuine weakness on your own terms, with time to fix it, instead of during a customer-facing outage during peak order volume at a company like Swiggy.

Security

Never run chaos experiments in production before validating the same experiment in staging. This project builds the practice safely on a local cluster — treat that discipline as non-negotiable before ever pointing these tools at a live environment.

Chaos engineering requires something with redundancy to test against. A single-replica deployment can't teach you anything useful — killing its only pod is simply an outage, not an experiment. Three replicas is the practical minimum for a meaningful pod-kill test.

Common Mistake

Running pod-kill experiments against a single-replica deployment isn't a chaos experiment — it's just an outage. Chaos engineering requires redundancy to actually test against.

Chaos Mesh is a CNCF project providing Kubernetes-native chaos engineering. Instead of writing ad-hoc scripts to kill processes, you apply YAML resources — PodChaos, NetworkChaos, StressChaos — that Chaos Mesh translates into real, time-bounded failures.

Access the dashboard:

Open http://localhost:2333 in your browser.

Tip

The Chaos Mesh dashboard lets you build and launch experiments visually, which is a fast way to explore what's possible — but writing the YAML directly, as we do in this project, is what you'll actually check into Git for repeatable experiments.

k6 generates realistic HTTP traffic during the experiment so you're measuring the failure's effect on real simulated load, not a theoretical availability number. The thresholds block in the script below encodes your SLOs directly — k6 will report pass/fail against them automatically.

Let it run to steady state before injecting any chaos — this baseline run is what every experiment result gets compared against.

Tip

Keep this port-forward and k6 run active in a separate terminal tab throughout Milestones 4-6 — every chaos experiment needs continuous traffic running alongside it to be measurable at all.

Hypothesis: When one of three orders-api pods is killed, the remaining two pods absorb the traffic without the error rate exceeding 1% or p99 latency exceeding 200ms.

Writing the hypothesis before running the experiment matters — it forces you to commit to a measurable prediction instead of retroactively deciding what the result "meant."

Watch the k6 output during the experiment for error spikes or latency increases during the restart window.

Common Mistake

Not watching k6's live output during the experiment window and only checking the final summary. The interesting data is the transient spike during the 10-30 second restart window — the final summary can average it away and hide exactly what you're trying to observe.

Hypothesis: When 300ms of network delay is injected on outbound traffic from the orders-api pods, users will observe latency increases but the service will not return errors.

This experiment simulates a slow downstream dependency — the kind of thing that happens when a payment gateway or a third-party API (a real scenario for a company like Razorpay) degrades without actually going down.

Watch k6's p99 latency metric during this run — the question isn't whether it rises, but whether it stays under your SLO threshold.

Tip

Try adjusting latency to 800ms and re-running this same experiment. Finding the exact delay value where your SLO actually breaches gives you a concrete number for capacity planning conversations, instead of a vague sense that "slow downstreams are bad."

Hypothesis: When CPU is throttled to 50% of the container limit across all pods, the service will maintain its error rate SLO but may breach the latency SLO.

This experiment tests a different failure mode entirely — not a dependency slowing down, but the pods themselves running under resource pressure, which is common during traffic spikes like a Zerodha market-open surge.

The experiment cleans itself up automatically after 2 minutes.

Common Mistake

Not setting an explicit duration on any chaos experiment. Without it, an experiment like this one persists until someone manually deletes it — a forgotten Friday-afternoon experiment can quietly degrade a system all weekend. Every experiment YAML in this project sets one deliberately.

Documenting results is as important as running the experiment — an experiment without a written finding gets forgotten by the next incident, and the whole point of chaos engineering is to retain what you learn.

Tip

Store chaos-report.md in the same Git repo as your Kubernetes manifests. Over time, a folder of these reports becomes a searchable history of every weakness your team has already found and fixed — genuinely useful during onboarding or post-incident reviews.

A Pod Disruption Budget tells Kubernetes the minimum number of pods that must remain available during any voluntary disruption (like a node drain or, in this case, Chaos Mesh's pod-kill action). It's the direct fix for the weakness Experiment 1 uncovered.

Re-run Experiment 1 (Milestone 4) with the PDB in place. The pod kill should now cause noticeably less disruption, since Kubernetes will enforce the minimum available pod count during any voluntary disruption.

Tip

A PDB only protects against voluntary disruptions like this one — it does nothing against an involuntary failure like a node hardware crash. Don't mistake "passed the pod-kill experiment" for "fully resilient to all pod loss."

Tip

Compare the k6 threshold pass/fail output (not just eyeballing the numbers) between the first and second pod-kill runs — k6 reports whether your defined thresholds passed or failed for that run, giving you an objective before/after comparison.