Your service has three replicas. Your dashboards are green. Everyone assumes that if one pod dies, the other two absorb the traffic without anyone noticing. Nobody has ever actually tested that assumption. Then one night, a node gets drained for a routine upgrade. The pod that dies happens to be the one holding a stale connection pool. The other two replicas were already running close to their connection limit. The service falls over, and the team spends forty minutes discovering something that a five-minute experiment in staging would have shown a month earlier. This is the entire premise of **chaos engineering**. Every distributed system contains assumptions nobody has verified. Chaos engineering finds them on a Tuesday afternoon with an engineer watching a dashboard, instead of finding them during an actual outage with a customer watching a broken app. As covered in the distributed systems module, cascading failures and retry storms are theoretical failure modes until you have actually watched one happen. Chaos engineering is how you watch one happen on your own terms. > 📌 **Remember:** Chaos engineering does not create fragility. It reveals fragility > that was already there. The system was never as reliable as the green dashboard > suggested - chaos just makes that visible before a real incident does.
Before you break anything, you need a precise definition of "working". This is the **steady state hypothesis** - a measurable, specific statement of what normal looks like, expressed in the same metrics you already use for SLOs. A vague hypothesis like "the service should stay up" is useless because you cannot tell if it passed or failed. A good hypothesis is falsifiable and specific: "We believe p99 latency for the checkout API stays below 500ms and error rate stays below 0.5% when one payments-service pod out of three is killed." ### Why you must define this before touching anything Without a steady state hypothesis, chaos engineering becomes what practitioners call chaos tourism - randomly breaking things and eyeballing whether it "feels okay". You cannot detect a 15% latency regression by eyeballing a dashboard. You can detect it by comparing a specific number against a specific threshold you wrote down in advance. ### Why observability comes first You cannot run chaos experiments on a system with no SLOs and no dashboards. If you do not know what normal p99 latency is, you cannot tell whether killing a pod made it worse. This is why chaos engineering is a Section 9 topic - it depends on the observability and SLO work covered earlier in this roadmap. > 🔴 **Common Mistake:** Teams run their first chaos experiment directly in production > before proving the approach works in staging. Start in staging, build confidence in > your tooling and your rollback process, then move to production with a small blast > radius. Production chaos without a staging track record is how you turn a controlled > experiment into an actual incident.
**Blast radius** is the scope of what your experiment can affect - how many pods, how many services, how many users. Every chaos experiment must have its blast radius decided before the experiment starts, not discovered during it. Start small and expand only after you have confidence: Week 1: 1 pod, staging, off-peak hours Week 3: 1 pod, production, off-peak hours Week 6: 1 AZ, production, business hours Week 12: automated nightly chaos in staging * This progression matters more than the tools. A team that jumps straight to killing an entire availability zone in production on day one is not doing chaos engineering - it is doing an incident with extra steps. * Always have a stop button. Every experiment needs a way to abort immediately if the steady state hypothesis is violated in a way that threatens real users. > ⚠️ **Security:** Never run an unbounded chaos experiment against production without > an automatic abort condition tied to your SLO dashboards. A NetworkPartition > experiment with no timeout and no kill switch can turn a five-minute test into a > multi-hour outage if the person running it steps away.
A **Game Day** is a scheduled team exercise where the SRE team deliberately triggers a failure scenario and runs the full incident response process against it - not just testing whether the system survives, but testing whether the humans and the runbooks do too. This is the difference between a chaos experiment and a Game Day. A chaos experiment tests the system. A Game Day tests the system and the response process together, using the same incident commander structure and communication cadence covered in the incident management module. ### What a Game Day actually looks like 1. Pick a scenario in advance - "primary database pod is killed during business hours" 2. Do not tell the on-call engineer exactly when it will happen (but do tell them a Game Day is scheduled that week - this is not a surprise attack on morale) 3. Trigger the failure using Chaos Mesh or LitmusChaos 4. Run the real incident process - declare severity, assign an incident commander, use the real Slack channels 5. Resolve it, then write a Game Day report - not a postmortem, but structured the same way: what worked, what surprised the team, what runbook was missing > 💡 **Tip:** The most valuable output of a Game Day is usually not "the system > handled it fine" - it is discovering that the runbook links to a Grafana dashboard > that was deleted six months ago, or that nobody remembers the Slack command to page > the database team.
Teams do not start with automated production chaos. The **Chaos Maturity Model** describes the realistic progression most SRE organisations follow. | Stage | What it looks like | Typical duration | |:---|:---|:---| | Ad-hoc | Manual pod kills, one engineer, staging only | First 1-2 months | | Structured | Documented hypotheses, scheduled Game Days | Months 2-6 | | Automated staging | Nightly chaos runs via LitmusChaos in CI | Months 6-12 | | Continuous production | Automated chaos with auto-abort in production | 12+ months | Trying to skip stages is the single most common reason chaos engineering initiatives fail. A team with no Game Day experience that tries to run continuous automated production chaos on month one will produce real incidents, not learning.
**Chaos Mesh** is a Kubernetes-native chaos engineering platform. It runs as a controller inside your cluster and lets you define experiments as Kubernetes custom resources - the same declarative model you already use for Deployments and Services. ### How it is built +------------------+ +-------------------+ | chaos-controller |------>| chaos-daemon | | -manager | | (per node, | | (schedules and | | DaemonSet) | | watches CRDs) | +---------+---------+ +------------------+ | v +-------------------+ | Target pod(s) | | fault injected | +-------------------+ * This diagram shows the two-part architecture: the controller-manager reads your experiment YAML and decides what to do, and the chaos-daemon running on every node actually performs the fault injection at the kernel or container runtime level. > **Note:** The chaos-daemon needs privileged access on each node because injecting > network delay or corrupting I/O requires manipulating things below the container > boundary - network interfaces, cgroups, syscalls. This is why Chaos Mesh > installation requires elevated RBAC permissions. ### Pod fault experiments * **PodKill** - immediately terminates the target pod, simulating a hard crash or node failure. Kubernetes reschedules it per your normal pod lifecycle. * **PodFailure** - makes the pod unavailable for a set duration without deleting it, simulating a hung or unresponsive process. * **ContainerKill** - kills one container inside a multi-container pod, useful for testing sidecar failure independently from the main application container. ```yaml apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: payments-pod-kill namespace: prod-mumbai spec: action: pod-kill mode: one ## target exactly one matching pod selector: namespaces: - prod-mumbai labelSelectors: app: payments-service scheduler: cron: "@every 1m" ## fire once, one minute after apply ``` > **Note:** `mode: one` picks a single random pod matching the selector. Other > options include `all` (every matching pod), `fixed` (an exact count), and > `fixed-percent` (a percentage of matching pods) - start with `one` until you trust > the blast radius. ### Network fault experiments * **NetworkDelay** - adds latency to traffic between services, testing whether timeouts and circuit breakers actually trigger the way you designed them to. * **NetworkPartition** - fully blocks traffic between two sets of pods, simulating a network split without killing anything. * **NetworkCorrupt** and **NetworkLoss** - corrupt or drop a percentage of packets, testing retry logic and idempotency under unreliable transport. ```yaml apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: checkout-to-payments-delay namespace: prod-mumbai spec: action: delay mode: all selector: labelSelectors: app: checkout-service delay: latency: "500ms" jitter: "50ms" ## small random variance so all requests aren't identical direction: to target: selector: labelSelectors: app: payments-service mode: all duration: "5m" ``` ### Stress and DNS fault experiments * **CPUStressor** and **MemoryStressor** - simulate a noisy neighbour node under resource pressure, testing whether your resource requests and limits actually protect your workload the way you assumed. * **DNS fault** - makes lookups fail for specific domains, testing whether your service degrades gracefully or hangs when DNS resolution breaks. * **IO fault** - injects disk latency or errors, testing whether your storage path handles slow or failing writes without corrupting data. ### Composing Chaos Workflows A single experiment tests one failure. Real incidents are rarely one thing - a **Chaos Workflow** chains multiple experiments in sequence to test compound failure, closer to how production actually breaks. Example: kill the database primary, then while the cluster is electing a new leader, inject 300ms of network delay on the follower - testing whether your failover path still meets its SLO under simultaneous degraded conditions, not just a clean failover.
Your service has three replicas. Your dashboards are green. Everyone assumes that if one pod dies, the other two absorb ...
Before you break anything, you need a precise definition of "working". This is the steady state hypothesis - a measurabl...
Blast radius is the scope of what your experiment can affect - how many pods, how many services, how many users. Every c...
A Game Day is a scheduled team exercise where the SRE team deliberately triggers a failure scenario and runs the full in...
Teams do not start with automated production chaos. The Chaos Maturity Model describes the realistic progression most SR...
Chaos Mesh is a Kubernetes-native chaos engineering platform. It runs as a controller inside your cluster and lets you d...
Where Chaos Mesh is built for interactive, one-off experiments run by an engineer at a keyboard, LitmusChaos is built fo...
You cannot chaos-test everything at once. FMEA - Failure Mode and Effects Analysis - is a systematic worksheet for decid...
This lab assumes a Kubernetes cluster with the payments-service example used throughout this roadmap, plus Prometheus an...
Tool / Concept Use it for Chaos Mesh PodKill Simulating hard crash or node failure Chaos Mesh NetworkDelay Testing timeo...
Running chaos experiments without SLOs defined first is the most common failure - without a steady state hypothesis tied...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.