Chaos Engineering Lab: Chaos Mesh, k6, Grafana

Run pod-kill, network-delay and CPU-stress experiments under live k6 load, watch the SLO impact in Grafana, write the report, then harden and re-test.

Architecture Overview & Problem Statement

This project builds a chaos engineering lab on Kubernetes: Chaos Mesh injects pod-kill, network-delay and CPU-stress failures into a three-replica service while k6 sends continuous load from inside the cluster and Grafana shows whether the service kept its SLO.

What you build, learn and can show

Build: a target service, a Prometheus and Grafana stack, an in-cluster k6 load test that writes to Prometheus, three Chaos Mesh experiments and a hardened redeploy
Learn: how to write a steady-state hypothesis, how to tell a real finding from a flawed experiment, and why a chaos tool's behaviour (for example, what it bypasses) matters as much as the YAML
Show: a repository with the experiment YAML, a Grafana dashboard and a written chaos report with measured before and after results

flowchart LR K[k6 load
in cluster] --> S[orders-api
3 pods] C[Chaos Mesh
experiment] -.->|injects failure| S K -->|remote write| P[Prometheus] P --> G[Grafana] classDef blue fill:#326CE5,color:#ffffff classDef orange fill:#F97316,color:#ffffff classDef green fill:#10B981,color:#ffffff classDef purple fill:#8B5CF6,color:#ffffff class K blue class S green class C orange class P,G purple

Every experiment follows the same five-step cycle, and the cycle is the real skill you are practising.

flowchart LR A[Steady state] --> B[Hypothesis] B --> C[Inject failure] C --> D[Observe] D --> E[Document] E --> F[Fix and re-test] classDef slate fill:#475569,color:#ffffff classDef orange fill:#F97316,color:#ffffff class A,B,D,E,F slate class C orange

The problem this solves

Most teams learn how their system fails during a real incident, at 2am, under user impact, with no way to isolate what broke. Netflix popularised the opposite habit: break things on purpose, in a controlled way, so you find weaknesses on your own schedule. A team at Hotstar might run exactly this practice before a high-traffic event like an IPL final.

You define a steady-state hypothesis, a measurable statement of normal backed by metrics. You inject one specific failure and check whether the steady state holds. A refuted hypothesis is a genuine weakness found with time to fix it.

Security
Never run chaos experiments in production before running the same experiment in staging. This project uses a disposable local or lab cluster. Treat that discipline as non-negotiable before pointing these tools at a live environment.

Prerequisites

A multi-node lab cluster (kind with 3 nodes works well) with kubectl and helm
Kubernetes 1.30 or newer if you want to use the preStop sleep action in Milestone 8
Experiments that use tc (network delay) need a cluster where the Chaos Mesh daemon can run privileged, which kind and most lab clusters allow

Milestone 1: Deploy the Target Application

This milestone deploys a three-replica service with a readiness probe, because chaos engineering needs redundancy to test against.

A single replica teaches nothing: killing its only pod is simply an outage. Three replicas is the practical minimum for a meaningful pod-kill test. The service is podinfo, a small open source Go app, so you do not have to build anything.

Bash

kubectl create namespace chaos-demo
 
cat > target-app.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: chaos-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: api
          image: ghcr.io/stefanprodan/podinfo:6.7.1
          ports:
            - name: http
              containerPort: 9898
          resources:
            requests:
              memory: "64Mi"
              cpu: "100m"
            limits:
              memory: "128Mi"
              cpu: "200m"
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            initialDelaySeconds: 3
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: orders-api
  namespace: chaos-demo
spec:
  selector:
    app: orders-api
  ports:
    - name: http
      port: 80
      targetPort: http
EOF
 
kubectl apply -f target-app.yaml
kubectl get pods -n chaos-demo

TEXT

NAME                          READY   STATUS
orders-api-7d9f8c6b-2kmpq     1/1     Running
orders-api-7d9f8c6b-5xnpl     1/1     Running
orders-api-7d9f8c6b-9rvbf     1/1     Running

Common Mistake
Running pod-kill against a single-replica deployment. That is an outage, not an experiment.

Milestone 2: Install Observability and Chaos Mesh

This milestone installs Prometheus and Grafana first, then Chaos Mesh, so every experiment can be measured.

Install observability before chaos. An experiment you cannot observe tells you nothing. The enableRemoteWriteReceiver setting lets k6 push its results straight into Prometheus.

Bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update
 
kubectl create namespace monitoring
 
export GRAFANA_PASSWORD="$(openssl rand -base64 18)"
echo "Grafana admin password: $GRAFANA_PASSWORD"
 
helm install kube-prometheus-stack \
    prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --set prometheus.prometheusSpec.enableRemoteWriteReceiver=true \
    --set grafana.adminPassword="$GRAFANA_PASSWORD" \
    --wait

Chaos Mesh is a CNCF project that gives Kubernetes native chaos resources (PodChaos, NetworkChaos, StressChaos). Instead of ad hoc scripts you apply YAML that Chaos Mesh turns into real, time-bounded failures.

Bash

kubectl create namespace chaos-mesh
 
## containerd is the runtime on kind and most managed clusters.
## Change these two values if your cluster uses something else.
helm install chaos-mesh chaos-mesh/chaos-mesh \
    --namespace chaos-mesh \
    --set chaosDaemon.runtime=containerd \
    --set chaosDaemon.socketPath=/run/containerd/containerd.sock \
    --wait
 
kubectl get pods -n chaos-mesh

TEXT

NAME                                  READY   STATUS
chaos-controller-manager-abc123-xyz   1/1     Running
chaos-daemon-2hkqp                    1/1     Running
chaos-daemon-5xnpl                    1/1     Running
chaos-dashboard-7d9f8c6b-klmnp        1/1     Running

Bash

kubectl port-forward -n chaos-mesh svc/chaos-dashboard 2333:2333 &
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80 &

Open http://localhost:2333 for Chaos Mesh and http://localhost:3000 for Grafana (user admin).

Tip
The Chaos Mesh dashboard is a fast way to explore what is possible, but writing the YAML yourself, as this project does, is what you commit to Git for repeatable experiments.

Milestone 3: Run k6 Inside the Cluster and Baseline the Service

This milestone runs a k6 load test as an in-cluster Job that writes metrics to Prometheus, then records a baseline every experiment is compared against.

Run k6 inside the cluster, not through kubectl port-forward. A port-forward to a Service binds to a single pod and does not load balance, so killing that one pod would break your test connection for reasons unrelated to the service. In-cluster traffic goes through the normal Service routing, which is what real users experience.

Bash

cat > load-test.js << 'EOF'
import http from 'k6/http';
import { check, sleep } from 'k6';
 
export const options = {
  stages: [
    { duration: '30s', target: 20 },
    { duration: '6m', target: 20 },
    { duration: '30s', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<200'],
  },
};
 
const TARGET = __ENV.TARGET;
 
export default function () {
  const res = http.get(`${TARGET}/`);
  check(res, {
    'status is 200': (r) => r.status === 200,
    'under 200ms': (r) => r.timings.duration < 200,
  });
  sleep(0.5);
}
EOF
 
kubectl create configmap k6-script -n chaos-demo --from-file=load-test.js

Bash

cat > k6-job.yaml << 'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: k6-load
  namespace: chaos-demo
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: k6
          image: grafana/k6:0.54.0
          args:
            - run
            - --out
            - experimental-prometheus-rw
            - --tag
            - testid=baseline
            - /scripts/load-test.js
          env:
            - name: TARGET
              value: "http://orders-api.chaos-demo.svc.cluster.local"
            - name: K6_PROMETHEUS_RW_SERVER_URL
              value: "http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/write"
            - name: K6_PROMETHEUS_RW_TREND_STATS
              value: "p(95),p(99),avg"
          volumeMounts:
            - name: script
              mountPath: /scripts
      volumes:
        - name: script
          configMap:
            name: k6-script
EOF
 
kubectl apply -f k6-job.yaml
kubectl logs -n chaos-demo job/k6-load -f

The baseline run is the reference for everything that follows. Note the baseline p99 latency and error rate from the end-of-test summary.

A Grafana dashboard for the experiments

k6 metric names can vary slightly between versions. Before building panels, open Grafana Explore and run {__name__=~"k6_.*"} to confirm the names below exist in your setup.

Bash

cat > chaos-dashboard.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: chaos-lab-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  chaos-lab.json: |
    {
      "uid": "chaos-lab",
      "title": "Chaos Lab: orders-api under load",
      "schemaVersion": 39,
      "time": {"from": "now-30m", "to": "now"},
      "panels": [
        {
          "id": 1, "type": "timeseries", "title": "p99 latency (ms)",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "k6_http_req_duration_p99", "legendFormat": "{{testid}}"}],
          "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}
        },
        {
          "id": 2, "type": "timeseries", "title": "Failed request rate",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "k6_http_req_failed_rate", "legendFormat": "{{testid}}"}],
          "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8}
        },
        {
          "id": 3, "type": "timeseries", "title": "Requests per second",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "sum by (testid) (rate(k6_http_reqs_total[30s]))", "legendFormat": "{{testid}}"}],
          "gridPos": {"x": 0, "y": 8, "w": 24, "h": 6}
        }
      ]
    }
EOF
 
kubectl apply -f chaos-dashboard.yaml

How to run each experiment

Every experiment uses the same routine so results are comparable. A Job's name cannot be reused, so delete it between runs.

Bash

## 1. Start a fresh load run tagged with the experiment name
kubectl delete job k6-load -n chaos-demo --ignore-not-found
sed 's/testid=baseline/testid=pod-kill/' k6-job.yaml | kubectl apply -f -
 
## 2. Wait about 2 minutes so the test is at steady state
## 3. Apply the chaos experiment (next milestones)
## 4. Watch the Grafana dashboard and the k6 logs
kubectl logs -n chaos-demo job/k6-load -f

Tip
Keep Grafana open on the chaos dashboard for every experiment. The interesting data is the short transient during the failure window, which the end-of-test summary can average away.

Milestone 4: Experiment 1: Pod Kill

This milestone kills one of three pods under load and checks whether the service keeps its error and latency targets.

Write the hypothesis before running the experiment. It forces you to commit to a measurable prediction instead of deciding afterwards what the result meant.

TEXT

Hypothesis: when one of three orders-api pods is killed, the remaining
pods absorb the traffic and the failed request rate stays below 1% and
p99 latency stays below 200ms.

Bash

cat > experiment-pod-kill.yaml << 'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-experiment
  namespace: chaos-demo
spec:
  action: pod-kill
  mode: one
  gracePeriod: 0
  selector:
    namespaces:
      - chaos-demo
    labelSelectors:
      app: orders-api
EOF
 
kubectl apply -f experiment-pod-kill.yaml
kubectl get pods -n chaos-demo --watch

gracePeriod: 0 makes the kill abrupt, like a crash. pod-kill is a one-shot action, so it needs no duration.

TEXT

NAME                     READY   STATUS
orders-api-7d9f-9rvbf    1/1     Running
orders-api-7d9f-9rvbf    0/1     Terminating
orders-api-7d9f-xk2lm    0/1     Pending
orders-api-7d9f-xk2lm    0/1     Running
orders-api-7d9f-xk2lm    1/1     Running

Record on the dashboard how long the failed request rate was above zero and the peak p99 latency, then clean up.

Bash

kubectl delete -f experiment-pod-kill.yaml

Common Mistake
Reading only the final k6 summary. The signal is a few seconds long and the summary averages it away. Read the dashboard during the window.

Tip
A killed pod is removed from Service endpoints within a second or two, so requests already in flight to it are the ones that fail. That is why you often see a small, short error blip rather than a sustained outage.

Milestone 5: Experiment 2: Network Delay

This milestone injects 300ms of network delay and checks the effect on latency and errors.

This models a degraded network path. The tc rule delays every packet leaving the pod, responses to k6 included. To delay only calls to one dependency, add a target selector. Real examples of this failure shape are a payment gateway or a third-party API slowing down without going down, a scenario a company like Razorpay worries about.

TEXT

Hypothesis: with 300ms of delay on traffic leaving the pods, latency rises
and breaches the 200ms target, but the service returns no errors.

Bash

cat > experiment-network-delay.yaml << 'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: network-delay-experiment
  namespace: chaos-demo
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - chaos-demo
    labelSelectors:
      app: orders-api
  direction: to
  delay:
    latency: "300ms"
    jitter: "50ms"
    correlation: "25"
  duration: "3m"
EOF
 
kubectl delete job k6-load -n chaos-demo --ignore-not-found
sed 's/testid=baseline/testid=network-delay/' k6-job.yaml | kubectl apply -f -
 
## after about 2 minutes of steady state:
kubectl apply -f experiment-network-delay.yaml

Watch p99 latency on the dashboard. The question is not whether it rises, it will, but by how much and whether errors appear.

Bash

kubectl delete -f experiment-network-delay.yaml

Tip
Re-run with latency: "800ms", then with 1500ms. The kubelet readiness probe has a 1 second timeout by default, so at the high end probes start failing, pods leave the Service and a latency problem turns into an availability problem. Finding that threshold gives you a concrete number for capacity planning.

Milestone 6: Experiment 3: CPU Stress

This milestone adds CPU pressure inside the pods and checks whether the service holds its SLOs.

StressChaos does not throttle the container. It starts stress workers inside the target container's cgroup that compete with your application for the CPU limit. That is the realistic shape of a noisy neighbour or a traffic spike.

TEXT

Hypothesis: with two CPU stress workers at 80% load in each pod, the error
rate stays below 1%, but p99 latency may breach 200ms because the app
competes with the stressors for its 200m CPU limit.

Bash

cat > experiment-cpu-stress.yaml << 'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: cpu-stress-experiment
  namespace: chaos-demo
spec:
  mode: all
  selector:
    namespaces:
      - chaos-demo
    labelSelectors:
      app: orders-api
  stressors:
    cpu:
      workers: 2
      load: 80
  duration: "2m"
EOF
 
kubectl delete job k6-load -n chaos-demo --ignore-not-found
sed 's/testid=baseline/testid=cpu-stress/' k6-job.yaml | kubectl apply -f -
 
## after about 2 minutes of steady state:
kubectl apply -f experiment-cpu-stress.yaml

The experiment removes itself after two minutes.

Common Mistake
Leaving out duration on an experiment that supports it. Without it the experiment runs until someone deletes it, and a forgotten Friday experiment can quietly degrade a system all weekend. Every continuous experiment in this project sets one.

Milestone 7: Write a Chaos Engineering Report

This milestone records what each experiment actually showed, so the next engineer does not have to rediscover it.

Documenting results matters as much as running them. An experiment with no written finding gets forgotten by the next incident. The template below has blanks on purpose: fill in your measured values and do not copy numbers from anyone else's run.

Bash

cat > chaos-report.md << 'EOF'
## Chaos Engineering Report: orders-api
 
Date: YYYY-MM-DD
Cluster: (kind or managed, node count)
 
### Steady state (from the baseline run)
* Load: 20 virtual users
* Baseline p99 latency: ___ ms
* Baseline failed request rate: ___ %
* Targets: failed requests under 1%, p99 under 200ms
 
### Experiment 1: Pod kill (gracePeriod 0)
Hypothesis: failed requests stay under 1% and p99 stays under 200ms.
Measured: failed rate peaked at ___ % for ___ seconds, p99 peaked at ___ ms.
Result: CONFIRMED / REFUTED / PARTIALLY CONFIRMED
Finding: ___
Fix: ___
 
### Experiment 2: Network delay (300ms)
Hypothesis: latency breaches 200ms, no errors.
Measured: p99 peaked at ___ ms, failed rate ___ %.
Result: ___
Finding: ___
Fix: ___
 
### Experiment 3: CPU stress
Hypothesis: errors stay under 1%, latency may breach.
Measured: p99 peaked at ___ ms, failed rate ___ %.
Result: ___
Finding: ___
 
### Action items
 
| Finding | Fix | Priority |
| :--- | :--- | :--- |
| ___ | ___ | HIGH / MEDIUM / LOW |
EOF

How to judge a result: a hypothesis is confirmed only if every number you predicted held. If you predicted "latency breaches but no errors" and latency did breach with no errors, that is a confirmed hypothesis even though the SLO was violated. The finding is what to do about the violation.

Tip
Store chaos-report.md in the same repository as your manifests. Over time a folder of reports becomes a searchable history of weaknesses already found and fixed.

Milestone 8: Fix the Weakness Found and Re-test

This milestone hardens the deployment against what you observed, then re-runs the pod-kill experiment to compare before and after.

A PodDisruptionBudget (PDB) tells Kubernetes the minimum number of pods that must stay available during voluntary disruptions such as node drains. It does not stop Chaos Mesh pod-kill, which deletes pods directly. The levers that do help a crash are spare replicas, spreading pods across nodes so one node failure cannot take several, and clients that retry. A graceful termination (the kind a rolling update or drain performs) is helped by a preStop delay that gives load balancers time to stop sending traffic.

Bash

cat > target-app-hardened.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: chaos-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: orders-api
      containers:
        - name: api
          image: ghcr.io/stefanprodan/podinfo:6.7.1
          ports:
            - name: http
              containerPort: 9898
          lifecycle:
            preStop:
              sleep:
                seconds: 5
          resources:
            requests:
              memory: "64Mi"
              cpu: "100m"
            limits:
              memory: "128Mi"
              cpu: "200m"
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            initialDelaySeconds: 3
            periodSeconds: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api-pdb
  namespace: chaos-demo
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: orders-api
EOF
 
kubectl apply -f target-app-hardened.yaml
kubectl rollout status deployment/orders-api -n chaos-demo

Re-run the experiment two ways

Repeat Experiment 1 with the same load and compare. Then run a graceful variant, where the kill honours the grace period and the preStop delay.

Bash

cat > experiment-pod-kill-graceful.yaml << 'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-graceful
  namespace: chaos-demo
spec:
  action: pod-kill
  mode: one
  gracePeriod: 30
  selector:
    namespaces:
      - chaos-demo
    labelSelectors:
      app: orders-api
EOF
 
kubectl delete job k6-load -n chaos-demo --ignore-not-found
sed 's/testid=baseline/testid=pod-kill-graceful/' k6-job.yaml | kubectl apply -f -
 
## after about 2 minutes of steady state:
kubectl apply -f experiment-pod-kill-graceful.yaml

Compare the abrupt and graceful runs on the dashboard. Put both measured results, and the before and after for abrupt kill, in your report.

See what a PDB actually protects

On a multi-node cluster, drain a node that hosts an orders-api pod. The drain uses the eviction API, so the PDB applies.

Bash

kubectl get pods -n chaos-demo -o wide
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
## Evictions that would drop below minAvailable 2 are retried until a replacement is ready
kubectl uncordon <node-name>

Tip
Passing the pod-kill experiment does not mean the service is resilient to all pod loss. A PDB does nothing against an involuntary failure such as a node hardware crash. Say that plainly in your report.

Validation & Testing

This final step confirms the lab works end to end and that nothing is left running.

Bash

## 1. Metrics flow from k6 into Prometheus
## In Grafana Explore, run: {__name__=~"k6_.*"}
## Expect a list of k6_* series
 
## 2. The PDB is active after hardening
kubectl get pdb -n chaos-demo
## Expect orders-api-pdb with MIN AVAILABLE 2
 
## 3. Hardened pods are spread across nodes
kubectl get pods -n chaos-demo -o wide
## Expect pods on more than one node when you have several nodes
 
## 4. Re-run the abrupt pod-kill under load and compare to Milestone 4
kubectl apply -f experiment-pod-kill.yaml
## Compare failed-request peak and duration on the dashboard
kubectl delete -f experiment-pod-kill.yaml
 
## 5. No experiments are left running
kubectl get podchaos,networkchaos,stresschaos -n chaos-demo
## Expect: No resources found
 
## 6. Your report has measured numbers, not blanks
grep -n "___" chaos-report.md
## Expect no output

What to commit

target-app.yaml, target-app-hardened.yaml, k6-job.yaml, load-test.js
The three experiment YAML files plus the graceful variant
chaos-dashboard.yaml and a screenshot of one experiment on the dashboard
The completed chaos-report.md

Common mistakes recap

Mistake	Why it breaks	Fix
No observability before chaos	You cannot tell whether the experiment did anything	Install Prometheus and Grafana first
Load through `kubectl port-forward`	A port-forward pins one pod and hides load balancing	Run k6 inside the cluster
Assuming a PDB stops pod-kill	PDBs only apply to evictions, not direct deletes	Use spare replicas, spreading and retries, and test PDBs with a drain
No `duration` on a continuous experiment	It persists until deleted	Always set one
Single-replica pod-kill	It is an outage, not an experiment	Require redundancy first
Writing the hypothesis afterwards	It defeats the purpose	Hypothesis first, always
Copying someone else's numbers into your report	You learn nothing about your own system	Fill the template from your own dashboard

Found a mistake, an outdated step or a broken link?

Chaos Engineering Lab: Chaos Mesh, k6, Grafana

Blueprint Walkthrough

Architecture Overview & Problem Statement

What you build, learn and can show

The problem this solves

Prerequisites

Milestone 1: Deploy the Target Application

Milestone 2: Install Observability and Chaos Mesh

Milestone 3: Run k6 Inside the Cluster and Baseline the Service

A Grafana dashboard for the experiments

How to run each experiment

Milestone 4: Experiment 1: Pod Kill

Milestone 5: Experiment 2: Network Delay

Milestone 6: Experiment 3: CPU Stress

Milestone 7: Write a Chaos Engineering Report

Milestone 8: Fix the Weakness Found and Re-test

Re-run the experiment two ways

See what a PDB actually protects

Validation & Testing

What to commit

Common mistakes recap