Build an End-to-End SRE Platform on Kubernetes

Build an SLO-driven reliability platform on Kubernetes: burn-rate alerts, a CI gate that auto-rolls back, chaos tests, DORA metrics, one portfolio repo.

Architecture Overview & Problem Statement

This project builds an SLO-driven reliability platform on Kubernetes: Prometheus burn-rate alerts watch a service, a GitHub Actions deployment gate rolls back a release automatically when the SLO is breached, Chaos Mesh tests the assumptions behind that SLO, and DORA metrics show whether all this automation is making the team faster and safer.

What you build, learn and can show

Build: a sample orders-api service, SLI recording rules, multi-window burn-rate alerts, a deploy pipeline with an SLO gate and automatic rollback, a chaos experiment, a DORA dashboard, a postmortem, a runbook and an ADR
Learn: how to turn an SLI into an error budget, how burn-rate alerting avoids both noisy and late pages, why a deployment gate must fail safely, and how DORA metrics are measured from real pipeline events
Show: one GitHub repository with a recorded demo of a bad release being rolled back without a human touching it

flowchart LR A[Git push] --> B[GitHub Actions
deploy] B --> C[orders-api
production] C -->|metrics| D[Prometheus] D --> E{SLO gate} E -->|breach| F[Auto rollback] E -->|healthy| G[Release kept] D --> H[Grafana
SLO and DORA] I[Chaos Mesh] -.->|injects failure| C classDef blue fill:#326CE5,color:#ffffff classDef green fill:#10B981,color:#ffffff classDef orange fill:#F97316,color:#ffffff classDef red fill:#EF4444,color:#ffffff classDef purple fill:#8B5CF6,color:#ffffff class A,B blue class C,D,H green class E purple class F red class G green class I orange

The problem this solves

Monitoring, chaos engineering and CI/CD each solve a narrow problem. Deployed side by side they still leave a gap: nothing uses the SLO data to make a decision. A team at a company like Razorpay can have a Grafana dashboard showing an SLO breach and still ship the next release five minutes later, because the pipeline never reads that signal.

This project closes the gap. The SLO stack feeds a deployment gate that halts and reverts a release if the service breaches its SLO in the minutes after deploy. Each chaos experiment tests one assumption the error budget policy depends on. DORA metrics close the loop by telling you objectively whether the automation is helping.

Remember
Any figures in this project (alert thresholds, burn-rate multipliers, settle times) are starting targets to measure and tune against your own traffic, not guaranteed results.

Prerequisites

A Kubernetes cluster with at least 3 nodes worth of capacity (kind, minikube or a managed cluster) and kubectl plus helm installed
metrics-server installed if you want the HPA to work
A GitHub repository for the project. The pipeline needs to reach your cluster, so use a managed cluster or a self-hosted runner on the machine running kind. The SLO gate script runs anywhere kubectl works, so you can test it locally first.

Milestone 1: Understand the Core Concepts Before You Build

This milestone gives you the vocabulary every later milestone depends on: SLIs, SLOs, error budgets, DORA metrics and blameless postmortems.

A Service Level Indicator (SLI) is a precise, measurable definition of "good", for example "an HTTP response whose status code is not 5xx". A Service Level Objective (SLO) is a target for that indicator over a window, for example "99.5% of requests are good over a rolling 28 days". The gap between 100% and the target is the error budget: the unreliability you are explicitly allowed to spend before you stop shipping features.

Why error budgets change behaviour

An error budget turns a vague argument ("is the system too unstable to deploy today?") into a number anyone can check. An error budget policy, a table mapping budget consumed to allowed actions, is what turns the SLO from a metric into governance.

DORA metrics

DORA metrics measure delivery health above any single service: deployment frequency, lead time for changes, change failure rate and time to restore service. Read them together. High frequency with a high failure rate means you ship fast and break things. Low everything means the team is not shipping enough to learn.

Blameless postmortems

"Human error" is where a shallow investigation stops, not a root cause. A blameless postmortem keeps asking why until it reaches a missing test, alert or piece of automation, something the system failed to catch.

Tip
Before moving on, work out the error budget for a 99.9% SLO over 28 days in minutes (28 x 24 x 60 x 0.001). You will do the same sum for 99.5% in Milestone 3.

Milestone 2: Set Up the Project Structure

This milestone creates the repository layout so every later file has an obvious home.

Bash

mkdir sre-platform && cd sre-platform
 
mkdir -p \
    .github/workflows \
    app \
    kubernetes/namespaces \
    kubernetes/app \
    kubernetes/monitoring \
    kubernetes/chaos \
    kubernetes/slos \
    scripts \
    postmortems \
    runbooks \
    docs/adr
 
git init
cat > .gitignore << 'EOF'
*.env
secrets/
kubeconfig*
EOF

◈ DIAGRAM

sre-platform/
    .github/workflows/   <- the deploy pipeline (GitHub only runs workflows from here)
    app/                 <- Dockerfile for the sample service
    kubernetes/
        namespaces/      <- namespace definitions
        app/             <- orders-api, Service, ServiceMonitor, load generator
        monitoring/      <- Pushgateway values, Grafana dashboard
        chaos/           <- Chaos Mesh experiments
        slos/            <- PrometheusRule resources
    scripts/             <- the SLO gate
    postmortems/         <- blameless postmortems
    runbooks/            <- executable runbooks
    docs/                <- SLO spec, DORA definitions, ADRs

Milestone 3: Define SLOs Before Deploying Anything

This milestone writes the SLO specification first, so the numbers in the alert rules, the gate and the dashboard all come from one document.

Write the SLO before building anything to measure it. The error budget policy is the piece that later decides whether a deploy may proceed.

Bash

cat > docs/slo-specification.md << 'EOF'
## SLO Specification: Orders API
 
Version: 1.0
Owner: platform-team
 
### SLI definitions
 
Availability SLI
* Good event: HTTP response with a status code that is not 5xx
* Total events: all HTTP requests received by orders-api
 
Latency SLI
* Good event: HTTP response delivered in 250ms or less
* Total events: all HTTP requests received
* Why 250ms: the SLI threshold must match a histogram bucket boundary
  exposed by the service (le="0.25"), otherwise it cannot be measured exactly
 
### SLO targets
 
| SLO | Target | Window | Error budget |
| :--- | :--- | :--- | :--- |
| Availability | 99.5% | 28 days rolling | 0.5% of requests, about 3.4 hours of full outage |
| Latency | 95% of requests under 250ms | 28 days rolling | 5% of requests |
 
Budget maths: 28 days x 24 hours x 0.005 = 3.36 hours.
 
### Error budget policy
 
| Budget consumed | Action |
| :--- | :--- |
| 0% to 50% | Normal velocity. Deploy freely. |
| 50% to 75% | Reduce deploy frequency. Prioritise reliability work. |
| 75% to 100% | Freeze feature deploys. Reliability work only. |
| Over 100% | Postmortem required. No deploys until the cause is fixed. |
 
### Alert thresholds (multi-window burn rate)
 
Burn rate 1 means spending the budget exactly over the 28-day window.
Both windows must exceed the threshold, which stops short blips paging anyone.
 
| Alert | Long window | Short window | Burn rate | Budget spent in the long window | Action |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Critical | 1h | 5m | 14.4x | 2% | Page on-call |
| High | 6h | 30m | 6x | 5% | Slack to team channel |
| Ticket | 3d | 6h | 1x | 10% | Ticket for next sprint |
EOF

Remember
The error budget policy is not paperwork. In Milestone 6 the pipeline gate enforces a version of it automatically.

Milestone 4: Deploy the Application and SLO Stack

This milestone deploys a real service that exposes request metrics, installs Prometheus and Grafana, and loads the SLI recording rules and burn-rate alerts.

The sample service is podinfo, a small open source Go app that exposes Prometheus metrics (http_requests_total with a status label and a request duration histogram) and can be told to return random errors, which is how you will simulate a bad release.

Namespaces and the service

Bash

cat > kubernetes/namespaces/namespaces.yaml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: production
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: v1
kind: Namespace
metadata:
  name: chaos-testing
EOF
 
kubectl apply -f kubernetes/namespaces/namespaces.yaml

Bash

cat > app/Dockerfile << 'EOF'
FROM ghcr.io/stefanprodan/podinfo:6.7.1
ENV PODINFO_UI_MESSAGE="orders-api"
EOF
 
cat > kubernetes/app/orders-api.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: production
  labels:
    app: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: api
          image: ghcr.io/stefanprodan/podinfo:6.7.1
          ports:
            - name: http
              containerPort: 9898
          resources:
            requests:
              memory: "64Mi"
              cpu: "100m"
            limits:
              memory: "128Mi"
              cpu: "200m"
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            initialDelaySeconds: 3
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 10
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: orders-api
  namespace: production
  labels:
    app: orders-api
spec:
  selector:
    app: orders-api
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders-api
  namespace: production
spec:
  selector:
    matchLabels:
      app: orders-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: orders-api
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
EOF

SLOs need traffic to measure. A small in-cluster load generator keeps the metrics alive so dashboards never show "No data".

Bash

cat > kubernetes/app/loadgen.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loadgen
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: loadgen
  template:
    metadata:
      labels:
        app: loadgen
    spec:
      containers:
        - name: curl
          image: curlimages/curl:8.10.1
          command:
            - sh
            - -c
            - "while true; do curl -s -o /dev/null http://orders-api.production.svc/; sleep 0.2; done"
          resources:
            requests:
              cpu: "20m"
              memory: "16Mi"
EOF

Install the monitoring stack

Two settings matter here. Retention must be longer than the 28-day SLO window or the budget panel will silently use less data than it claims, and the selector flags let Prometheus pick up your ServiceMonitors and rules without extra labels.

Bash

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
 
export GRAFANA_PASSWORD="$(openssl rand -base64 18)"
echo "Grafana admin password: $GRAFANA_PASSWORD"
 
helm install kube-prometheus-stack \
    prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --set prometheus.prometheusSpec.retention=35d \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false \
    --set grafana.adminPassword="$GRAFANA_PASSWORD" \
    --wait
 
kubectl apply -f kubernetes/app/orders-api.yaml
kubectl apply -f kubernetes/app/loadgen.yaml
kubectl rollout status deployment/orders-api -n production

Security
Never commit a Grafana password to Git, even in a lab. Generate it at install time as above and keep it out of the repository.

SLI recording rules and burn-rate alerts

Bash

cat > kubernetes/slos/orders-api-slos.yaml << 'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: orders-api-slo-rules
  namespace: monitoring
spec:
  groups:
    - name: orders-api-sli
      interval: 30s
      rules:
        # "or vector(0)" matters: with no 5xx series the numerator would be
        # empty and the whole ratio would return no data instead of 0.
        - record: orders_api:error_ratio:rate5m
          expr: |
            (sum(rate(http_requests_total{namespace="production",job="orders-api",status=~"5.."}[5m])) or vector(0))
            / sum(rate(http_requests_total{namespace="production",job="orders-api"}[5m]))
        - record: orders_api:error_ratio:rate30m
          expr: |
            (sum(rate(http_requests_total{namespace="production",job="orders-api",status=~"5.."}[30m])) or vector(0))
            / sum(rate(http_requests_total{namespace="production",job="orders-api"}[30m]))
        - record: orders_api:error_ratio:rate1h
          expr: |
            (sum(rate(http_requests_total{namespace="production",job="orders-api",status=~"5.."}[1h])) or vector(0))
            / sum(rate(http_requests_total{namespace="production",job="orders-api"}[1h]))
        - record: orders_api:error_ratio:rate6h
          expr: |
            (sum(rate(http_requests_total{namespace="production",job="orders-api",status=~"5.."}[6h])) or vector(0))
            / sum(rate(http_requests_total{namespace="production",job="orders-api"}[6h]))
        - record: orders_api:error_ratio:rate3d
          expr: |
            (sum(rate(http_requests_total{namespace="production",job="orders-api",status=~"5.."}[3d])) or vector(0))
            / sum(rate(http_requests_total{namespace="production",job="orders-api"}[3d]))
        - record: orders_api:error_ratio:rate28d
          expr: |
            (sum(rate(http_requests_total{namespace="production",job="orders-api",status=~"5.."}[28d])) or vector(0))
            / sum(rate(http_requests_total{namespace="production",job="orders-api"}[28d]))
        - record: orders_api:latency_good_ratio:rate5m
          expr: |
            sum(rate(http_request_duration_seconds_bucket{namespace="production",job="orders-api",le="0.25"}[5m]))
            / sum(rate(http_request_duration_seconds_count{namespace="production",job="orders-api"}[5m]))
 
    - name: orders-api-slo-alerts
      rules:
        - alert: OrdersAPIBurnRateCritical
          expr: |
            orders_api:error_ratio:rate1h > (14.4 * 0.005)
            and
            orders_api:error_ratio:rate5m > (14.4 * 0.005)
          for: 2m
          labels:
            severity: critical
            slo: orders-api-availability
          annotations:
            summary: "orders-api is burning error budget at 14.4x"
            description: "1h error ratio is {{ $value | humanizePercentage }}. At this rate 2% of the 28-day budget is spent every hour and the whole budget lasts about 47 hours."
        - alert: OrdersAPIBurnRateHigh
          expr: |
            orders_api:error_ratio:rate6h > (6 * 0.005)
            and
            orders_api:error_ratio:rate30m > (6 * 0.005)
          for: 15m
          labels:
            severity: warning
            slo: orders-api-availability
          annotations:
            summary: "orders-api is burning error budget at 6x"
            description: "6h error ratio is {{ $value | humanizePercentage }}. 5% of the budget is spent every 6 hours."
        - alert: OrdersAPIBurnRateSlow
          expr: |
            orders_api:error_ratio:rate3d > (1 * 0.005)
            and
            orders_api:error_ratio:rate6h > (1 * 0.005)
          for: 1h
          labels:
            severity: ticket
            slo: orders-api-availability
          annotations:
            summary: "orders-api is spending budget faster than the SLO allows"
            description: "3d error ratio is {{ $value | humanizePercentage }}. Create a ticket for the next sprint."
EOF
 
kubectl apply -f kubernetes/slos/orders-api-slos.yaml

Common Mistake
Deploying the app without readiness probes, a PodDisruptionBudget and several replicas. The rollback in Milestone 6 only helps if the deployment can recover cleanly once traffic shifts back.

Tip
Port-forward Prometheus (kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring) and run orders_api:error_ratio:rate5m. A flat 0 means the pipeline from app to rule is working.

Milestone 5: Run a Chaos Experiment to Validate the SLO Assumptions

This milestone installs Chaos Mesh and checks that killing one of three pods does not breach the availability SLO, before you trust the automated gate with real decisions.

The hypothesis: with three replicas and a readiness probe, killing one pod keeps the 5-minute error ratio below 0.5%.

Bash

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update
 
kubectl create namespace chaos-mesh
 
## Use the runtime and socket your cluster actually has.
## kind and most managed clusters use containerd, shown here.
helm install chaos-mesh chaos-mesh/chaos-mesh \
    --namespace chaos-mesh \
    --set chaosDaemon.runtime=containerd \
    --set chaosDaemon.socketPath=/run/containerd/containerd.sock \
    --wait

Bash

cat > kubernetes/chaos/pod-kill-validation.yaml << 'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: sre-platform-pod-kill
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: orders-api
EOF
 
kubectl get pods -n production -w &
kubectl apply -f kubernetes/chaos/pod-kill-validation.yaml

pod-kill is a one-shot action, so it needs no duration. Wait a few minutes, then query Prometheus for the worst value in the window and clean up.

Bash

## In the Prometheus UI, run:
##   max_over_time(orders_api:error_ratio:rate5m[10m])
## Hypothesis holds if the result is below 0.005
 
kubectl delete -f kubernetes/chaos/pod-kill-validation.yaml

Common Mistake
Assuming the PodDisruptionBudget protects you from Chaos Mesh pod-kill. A PDB only limits evictions (node drains and similar). A direct pod delete bypasses it. What actually keeps the SLO intact here is having spare replicas and a correct readiness probe.

Tip
If the hypothesis is refuted, that is a finding, not a failure. Record the measured value and what you changed in docs/ before moving on.

Milestone 6: Build the Automated Reliability Pipeline

This milestone builds a GitHub Actions pipeline that builds an image, deploys it, waits for metrics to settle, queries Prometheus through the Kubernetes API, and rolls back automatically if the SLO is breached.

The gate lives in a script, not inline YAML, so you can run it by hand against a bad release and unit test it. It has three outcomes: ok, breach (roll back), and unknown (no data, fail the job but do not roll back, since a Prometheus outage should not revert a healthy release).

The SLO gate script

Bash

cat > scripts/slo_gate.py << 'EOF'
#!/usr/bin/env python3
"""
SLO gate for orders-api.
 
Usage: python3 scripts/slo_gate.py [threshold]
 
Queries Prometheus through the Kubernetes API server proxy, so it works from
any machine that has kubectl access (including a GitHub-hosted runner).
 
Exit codes: 0 = within SLO, 1 = breach (roll back), 2 = no data (do not roll back)
"""
import json
import os
import subprocess
import sys
import urllib.parse
 
THRESHOLD = float(sys.argv[1]) if len(sys.argv) > 1 else 0.005
PROM_PROXY = (
    "/api/v1/namespaces/monitoring/services/"
    "kube-prometheus-stack-prometheus:9090/proxy/api/v1/query"
)
 
 
def query(expr):
    path = f"{PROM_PROXY}?query={urllib.parse.quote(expr)}"
    out = subprocess.run(
        ["kubectl", "get", "--raw", path],
        capture_output=True, text=True, check=True,
    ).stdout
    result = json.loads(out)["data"]["result"]
    return float(result[0]["value"][1]) if result else None
 
 
def emit(result):
    print(f"gate result: {result}")
    out_file = os.environ.get("GITHUB_OUTPUT")
    if out_file:
        with open(out_file, "a") as f:
            f.write(f"result={result}\n")
 
 
traffic = query('sum(rate(http_requests_total{namespace="production",job="orders-api"}[5m]))')
error_ratio = query("orders_api:error_ratio:rate5m")
 
if traffic is None or traffic == 0 or error_ratio is None:
    print("No traffic or no data: cannot judge the release")
    emit("unknown")
    sys.exit(2)
 
print(f"5m error ratio: {error_ratio:.4f} (threshold {THRESHOLD})")
if error_ratio > THRESHOLD:
    emit("breach")
    sys.exit(1)
 
emit("ok")
EOF
 
chmod +x scripts/slo_gate.py

Pushgateway for DORA events

The pipeline records every deploy as a metric so Milestone 7 can compute DORA numbers from real events. Install Pushgateway now so the pipeline can push to it.

Bash

helm install pushgateway prometheus-community/prometheus-pushgateway \
    --namespace monitoring \
    --set serviceMonitor.enabled=true \
    --set serviceMonitor.namespace=monitoring \
    --set serviceMonitor.additionalLabels.release=kube-prometheus-stack \
    --set persistentVolume.enabled=true \
    --wait

The workflow

GitHub only runs workflows stored in .github/workflows/. Store the kubeconfig for a service account with access to the production and monitoring namespaces as the repository secret KUBECONFIG, and make the GHCR package public so the cluster can pull the image.

Bash

cat > .github/workflows/deploy-with-slo-gate.yml << 'EOF'
name: Deploy with SLO gate
 
on:
  push:
    branches: [main]
    paths: [app/**]
 
permissions:
  contents: read
  packages: write
 
env:
  IMAGE: ghcr.io/${{ github.repository_owner }}/orders-api
 
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: app
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}
 
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Configure kubectl
        uses: azure/k8s-set-context@v4
        with:
          method: kubeconfig
          kubeconfig: ${{ secrets.KUBECONFIG }}
 
      - name: Deploy new version
        run: |
          kubectl set image deployment/orders-api \
              api=${{ env.IMAGE }}:${{ github.sha }} \
              -n production
          kubectl rollout status deployment/orders-api \
              -n production --timeout=300s
 
      # Checking straight after rollout measures deploy-time noise
      # (connection resets during the rolling update), not the new
      # version's steady state. Five minutes lets the 5m window clear.
      - name: Wait for metrics to settle
        run: sleep 300
 
      - name: SLO gate
        id: gate
        run: python3 scripts/slo_gate.py 0.005
 
      - name: Roll back on SLO breach
        if: always() && steps.gate.outputs.result == 'breach'
        run: |
          kubectl rollout undo deployment/orders-api -n production
          kubectl rollout status deployment/orders-api \
              -n production --timeout=300s
 
      - name: Notify Slack
        if: always() && steps.gate.outputs.result == 'breach'
        run: |
          curl -sS -X POST "${{ secrets.SLACK_WEBHOOK }}" \
            -H 'Content-type: application/json' \
            -d '{"text": ":rotating_light: AUTO-ROLLBACK orders-api. Commit ${{ github.sha }} breached the 0.5% error SLO."}'
 
      - name: Record deployment for DORA metrics
        if: always()
        run: |
          case "${{ steps.gate.outputs.result }}" in
            ok) RESULT=success ;;
            breach) RESULT=rolled_back ;;
            *) RESULT=failed ;;
          esac
          NOW=$(date +%s)
          COMMIT_TS=$(git show -s --format=%ct "${{ github.sha }}")
          LEAD=$((NOW - COMMIT_TS))
          printf 'deploy_result_timestamp_seconds{result="%s"} %s\ndeploy_lead_time_seconds %s\n' \
              "$RESULT" "$NOW" "$LEAD" > /tmp/metrics.txt
          kubectl run "pgw-push-$RANDOM" -n monitoring --rm -i --restart=Never \
              --image=curlimages/curl:8.10.1 -- \
              curl -sS --data-binary @- \
              "http://pushgateway-prometheus-pushgateway.monitoring:9091/metrics/job/deploy/instance/${{ github.sha }}" \
              < /tmp/metrics.txt
EOF

Prove the gate works by hand

Simulate a bad release without waiting on CI: make podinfo return random errors, wait for the 5-minute window to fill, then run the gate.

Bash

kubectl set env deployment/orders-api PODINFO_RANDOM_ERROR=true -n production
kubectl rollout status deployment/orders-api -n production
 
## Wait about 6 minutes, then:
python3 scripts/slo_gate.py 0.005
echo "exit code: $?"
## Expect exit code 1 and "gate result: breach"
 
kubectl rollout undo deployment/orders-api -n production

Common Mistake
Checking the SLO immediately after kubectl rollout status returns. You measure the rolling update's own transient errors and trigger false rollbacks. The 300 second settle time is deliberate, and you should tune it against your own traffic.

Tip
The gate only sees availability. As a stretch goal, add orders_api:latency_good_ratio:rate5m to the script and fail the gate when it drops below 0.95.

Milestone 7: Build the DORA Metrics Dashboard

This milestone turns the deploy events your pipeline pushes into the four DORA metrics, next to the SLO error budget, on one Grafana dashboard.

DORA metrics answer a different question from the SLO stack. The SLO asks whether the service is healthy now. DORA asks whether your delivery process is getting faster and safer.

Bash

cat > docs/dora-metrics-definition.md << 'EOF'
## DORA Metrics: Measurement Definitions
 
Source of truth: one series per deploy, pushed by the pipeline to Pushgateway.
* deploy_result_timestamp_seconds{result="success|rolled_back|failed"}
* deploy_lead_time_seconds
 
### Deployment frequency
Count of deploys in the last 7 days.
PromQL: count(deploy_result_timestamp_seconds > (time() - 604800))
 
### Lead time for changes
Time from commit to verified production deploy (includes the 5 minute settle and gate).
PromQL: avg(deploy_lead_time_seconds)
 
### Change failure rate
Share of deploys in 30 days that were rolled back or failed.
PromQL: (count(deploy_result_timestamp_seconds{result!="success"} > (time() - 2592000)) or vector(0))
        / count(deploy_result_timestamp_seconds > (time() - 2592000))
 
### Time to restore (MTTR)
Measured from alert firing to resolved. The dashboard shows total hours
spent with an SLO alert firing; divide by the number of incidents recorded
in postmortems/ to get MTTR.
PromQL: sum_over_time(ALERTS{alertname=~"OrdersAPIBurnRate.*",alertstate="firing"}[30d]) * 30 / 3600
        (30 is the evaluation interval in seconds)
 
### Reference targets (to compare against, not to promise)
Elite teams in the DORA research deploy on demand, with lead time under a day,
a change failure rate in the low single digits and recovery under an hour.
EOF

Grafana's sidecar loads any ConfigMap labelled grafana_dashboard: "1", so the dashboard is code in Git.

Bash

cat > kubernetes/monitoring/sre-dashboard.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: sre-platform-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  sre-platform.json: |
    {
      "uid": "sre-platform",
      "title": "SRE Platform: SLO and DORA",
      "schemaVersion": 39,
      "time": {"from": "now-6h", "to": "now"},
      "panels": [
        {
          "id": 1, "type": "stat", "title": "Deploys (7 days)",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "count(deploy_result_timestamp_seconds > (time() - 604800)) or vector(0)"}],
          "gridPos": {"x": 0, "y": 0, "w": 6, "h": 4}
        },
        {
          "id": 2, "type": "stat", "title": "Lead time (minutes)",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "avg(deploy_lead_time_seconds) / 60"}],
          "gridPos": {"x": 6, "y": 0, "w": 6, "h": 4}
        },
        {
          "id": 3, "type": "stat", "title": "Change failure rate (30 days)",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "(count(deploy_result_timestamp_seconds{result!=\"success\"} > (time() - 2592000)) or vector(0)) / count(deploy_result_timestamp_seconds > (time() - 2592000))"}],
          "fieldConfig": {"defaults": {"unit": "percentunit"}},
          "gridPos": {"x": 12, "y": 0, "w": 6, "h": 4}
        },
        {
          "id": 4, "type": "stat", "title": "Hours with SLO alert firing (30 days)",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "sum_over_time(ALERTS{alertname=~\"OrdersAPIBurnRate.*\",alertstate=\"firing\"}[30d]) * 30 / 3600 or vector(0)"}],
          "gridPos": {"x": 18, "y": 0, "w": 6, "h": 4}
        },
        {
          "id": 5, "type": "gauge", "title": "Error budget remaining (28 days)",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "100 * (1 - orders_api:error_ratio:rate28d / 0.005)"}],
          "fieldConfig": {"defaults": {
            "unit": "percent", "min": 0, "max": 100,
            "thresholds": {"mode": "absolute", "steps": [
              {"color": "red", "value": null},
              {"color": "yellow", "value": 25},
              {"color": "green", "value": 50}
            ]}
          }},
          "gridPos": {"x": 0, "y": 4, "w": 8, "h": 8}
        },
        {
          "id": 6, "type": "timeseries", "title": "Availability (1h window)",
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "targets": [{"expr": "100 * (1 - orders_api:error_ratio:rate1h)", "legendFormat": "availability %"}],
          "gridPos": {"x": 8, "y": 4, "w": 16, "h": 8}
        }
      ]
    }
EOF
 
kubectl apply -f kubernetes/monitoring/sre-dashboard.yaml

Tip
Deploy a few times, including one bad release, before judging the dashboard. DORA panels read as empty until the pipeline has pushed real events.

Remember
The target is high deployment frequency and a low change failure rate together. Either number alone is misleading.

Milestone 8: Write a Blameless Postmortem

This milestone writes a complete postmortem for a fictional incident so you can practise reaching a systemic root cause instead of stopping at a person.

The incident below is an illustrative example. Replace the names and numbers with a real incident (for example the bad release you triggered in Milestone 6) when you have one.

Bash

cat > postmortems/2026-08-14-orders-api-latency-incident.md << 'EOF'
## Postmortem: Orders API timeouts after v2.1.3 (illustrative example)
 
Date: 2026-08-14
Severity: P2
Duration: 47 minutes (14:32 to 15:19 IST)
Author: example on-call engineer
 
### Summary
After release v2.1.3, about 35% of orders-api requests timed out with 504
errors for 47 minutes. Root cause: a migration added a column without an
index, and a new query filtering on that column caused full table scans
under load.
 
### Impact
* Average traffic: 15 requests per second
* Failed requests: 15 x 0.35 x 2,820 seconds, about 14,800
* Error budget consumed: 47 minutes x 35% = 16.5 minutes of full outage,
  against a 28-day budget of 3.36 hours (201.6 minutes) = about 8.2%
 
### Timeline (IST)
* 14:28 v2.1.3 deployed by the pipeline (this predates the SLO gate)
* 14:32 Burn-rate alert fires
* 14:35 On-call acknowledges
* 14:38 On-call correlates the alert with the 14:28 deploy
* 14:52 Slow query log confirms full table scans from the new query
* 14:55 Rollback started, but against orders-worker, a similarly named service
* 15:02 No change in error rate, wrong target noticed
* 15:05 Correct rollback of orders-api
* 15:14 Error rate back to normal
* 15:19 Alert resolves. Time to restore: 47 minutes
 
### Root cause
The v2.1.3 migration added a column to the 45M-row orders table with no index.
A query added in the same release filters on that column.
 
### Contributing factors
* No query performance check in CI
* Load test data had 1,000 rows, nothing like production volume
* Two services with near-identical names and no rollback target confirmation
* No staging environment with production-shaped data
 
### Five whys
* Why did requests time out? Queries did full table scans.
* Why? The filtered column had no index.
* Why? The migration added the column but not the index.
* Why was that not caught? CI has no slow query analysis.
* Why not? Nobody owns building it. This is the systemic cause.
 
### Action items
 
| Action | Owner | Due | Status |
| :--- | :--- | :--- | :--- |
| Run EXPLAIN on new queries in CI and fail on sequential scans | Dev team | 2026-08-21 | Open |
| Seed staging with production-shaped data | Platform team | 2026-08-28 | Open |
| Add the missing index | Database team | 2026-08-16 | Closed |
| Rename orders-worker to orders-processor | Platform team | 2026-08-19 | Open |
| Add rollback target confirmation to the runbook (Milestone 9) | Platform team | 2026-08-21 | Open |
| Replay this incident against the SLO gate to confirm it would have rolled back | Platform team | 2026-08-21 | Open |
 
### Lessons learned
* A rollback needs a confirmation step that shows the target deployment
* Tiny load-test datasets hide exactly the failures that matter
* The fix is automated slow-query detection in CI, not asking the on-call to be more careful
EOF

Common Mistake
Ending the Five Whys at "the engineer rolled back the wrong service". That stops at a person. Keep asking why until you reach a missing check, alert or piece of automation.

Milestone 9: Write an Executable Runbook

This milestone turns the investigation steps for the main alert into a script anyone paged for the first time can run, including a rollback that makes you confirm the target.

Bash

cat > runbooks/orders-api-high-error-rate.py << 'EOF'
#!/usr/bin/env python3
"""
Runbook: orders-api high error rate
 
Use when OrdersAPIBurnRateCritical fires.
Usage: python3 runbooks/orders-api-high-error-rate.py
"""
import subprocess
from datetime import datetime
 
NAMESPACE = "production"
DEPLOYMENT = "orders-api"
 
 
def run(args):
    result = subprocess.run(args, capture_output=True, text=True)
    return (result.stdout + result.stderr).strip()
 
 
def check_pods():
    print("\n[1/5] Pod status")
    out = run(["kubectl", "get", "pods", "-n", NAMESPACE, "-l", f"app={DEPLOYMENT}"])
    print(out)
    if "CrashLoopBackOff" in out or "Error" in out:
        print("  FINDING: pods in an error state, check logs next")
 
 
def check_recent_deploys():
    print("\n[2/5] Rollout history")
    print(run(["kubectl", "rollout", "history", f"deployment/{DEPLOYMENT}", "-n", NAMESPACE]))
    print("  ACTION: did an error start shortly after a recent revision?")
 
 
def check_logs():
    print("\n[3/5] Recent error log lines")
    logs = run(["kubectl", "logs", "-n", NAMESPACE, "-l", f"app={DEPLOYMENT}",
                "--tail=100", "--prefix=true"])
    errors = [line for line in logs.splitlines() if "error" in line.lower()][:20]
    print("\n".join(errors) if errors else "  no error lines in the last 100 per pod")
 
 
def check_resources():
    print("\n[4/5] Resource usage")
    print(run(["kubectl", "top", "pods", "-n", NAMESPACE, "-l", f"app={DEPLOYMENT}"]))
 
 
def rollback():
    print("\n[5/5] Rollback decision")
    print(f"Target: deployment/{DEPLOYMENT} in namespace {NAMESPACE}")
    typed = input(f"Type the deployment name ({DEPLOYMENT}) to confirm a rollback, or press Enter to skip: ").strip()
    if typed != DEPLOYMENT:
        print("Rollback skipped. Continue investigating.")
        return
    print(run(["kubectl", "rollout", "undo", f"deployment/{DEPLOYMENT}", "-n", NAMESPACE]))
    print("Rollback requested. Watch Grafana for recovery.")
 
 
if __name__ == "__main__":
    print(f"orders-api runbook started {datetime.now().isoformat()}")
    check_pods()
    check_recent_deploys()
    check_logs()
    check_resources()
    rollback()
    print("\nWrite a postmortem if this was a P1 or P2 incident.")
EOF
 
chmod +x runbooks/orders-api-high-error-rate.py

Tip
Add a runbook_url annotation to the critical alert in orders-api-slos.yaml pointing at this file, so the page links straight to the next action.

Milestone 10: Write an Architecture Decision Record

This milestone documents why you chose Prometheus, so a future engineer can see the trade-offs without re-arguing the decision.

An ADR records the reasoning, not just the result. Use your own real constraints. The example below shows the shape.

Bash

cat > docs/adr/ADR-001-prometheus-over-saas-metrics.md << 'EOF'
## ADR 001: Prometheus and Grafana over a SaaS metrics vendor
 
Date: 2026-08-14
Status: Accepted
Deciders: Platform team
 
### Context
We need metrics and SLO alerting for a Kubernetes cluster. Options: self-hosted
Prometheus with Grafana, or a SaaS observability vendor.
 
### Decision
Prometheus with Grafana.
 
### Reasons
* Cost: SaaS vendors bill per host or per metric plus add-ons. Model your own
  cost with the vendor's current pricing calculator before deciding.
* Data control: metrics stay inside our cluster, which matters for
  financial transaction data.
* Alerts as code: PrometheusRule resources live in Git and are reviewed like
  application code.
 
### Trade-offs
* SaaS vendors offer better out-of-the-box dashboards and APM tracing
* Prometheus needs in-house expertise to run reliably
* Built-in anomaly detection is weaker
 
### Consequences
* We build and maintain our own dashboards
* Retention is set to 35 days, enough for a 28-day SLO window. Longer
  retention needs Thanos or Cortex.
 
### Review date
Revisit in 12 months, or sooner if the cost of operating Prometheus exceeds
the cost of a managed alternative.
EOF

Validation & Testing

This final step proves every piece of the platform works and that the pieces actually talk to each other.

Bash

## 1. Reliability primitives exist
kubectl get deployment,pdb,hpa -n production
## Expect orders-api, orders-api-pdb and orders-api-hpa
 
## 2. Rules loaded and the SLI returns data
kubectl get prometheusrule -n monitoring
## In the Prometheus UI, orders_api:error_ratio:rate5m returns a value (0 is fine)
 
## 3. Chaos experiment keeps the SLO
kubectl apply -f kubernetes/chaos/pod-kill-validation.yaml
## Then check max_over_time(orders_api:error_ratio:rate5m[10m]) < 0.005
kubectl delete -f kubernetes/chaos/pod-kill-validation.yaml
 
## 4. Workflow is valid YAML
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/deploy-with-slo-gate.yml'))"
## No output means valid
 
## 5. Gate catches a bad release (the key test)
kubectl set env deployment/orders-api PODINFO_RANDOM_ERROR=true -n production
## wait about 6 minutes
python3 scripts/slo_gate.py 0.005; echo "exit: $?"
## Expect exit: 1
kubectl rollout undo deployment/orders-api -n production
 
## 6. Runbook runs end to end
python3 runbooks/orders-api-high-error-rate.py
## Press Enter at the rollback prompt to skip
 
## 7. Documents are committed
git add -A && git commit -m "SRE platform"
git log --oneline -- postmortems/ docs/adr/

What the finished repository gives you

A multi-replica service with PDB, HPA and readiness probes, measured by real SLI recording rules
Multi-window burn-rate alerts at 14.4x, 6x and 1x
A pipeline with an SLO gate and automatic rollback, plus a gate script you can run by hand
A Chaos Mesh experiment that tests the platform's own assumption
A DORA dashboard fed by real deploy events
A postmortem, an executable runbook and an ADR

Quick reference

Component	Access	Purpose
Grafana	`kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring`	SLO and DORA dashboards
Prometheus	`kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring`	Query raw metrics
Chaos dashboard	`kubectl port-forward svc/chaos-dashboard 2333:2333 -n chaos-mesh`	Manage experiments
Runbook	`python3 runbooks/orders-api-high-error-rate.py`	Guided incident response

Common mistakes

Building dashboards before the application. Deploy a real service with traffic first, or every panel says "No data" and proves nothing to a reviewer.
Treating DORA metrics as vanity numbers. Read all four together, as in Milestone 1.
Postmortems that stop at "human error". Every one should end at a missing test, alert or automation.
Gating on a ratio that returns no data when there are no errors. Use or vector(0) on the numerator, as the recording rules do.
Checking the SLO with no settle time, which causes false rollbacks.
Rolling back automatically when Prometheus itself is unreachable. Treat "no data" as a failed job, not as a breach.
Committing secrets or kubeconfigs. Keep them in GitHub Secrets and out of the repository.
Not recording a demo. A short video of a bad release being rolled back automatically convinces reviewers far more than a README.

Found a mistake, an outdated step or a broken link?

Build an End-to-End SRE Platform on Kubernetes

Blueprint Walkthrough

Architecture Overview & Problem Statement

What you build, learn and can show

The problem this solves

Prerequisites

Milestone 1: Understand the Core Concepts Before You Build

Why error budgets change behaviour

DORA metrics

Blameless postmortems

Milestone 2: Set Up the Project Structure

Milestone 3: Define SLOs Before Deploying Anything

Milestone 4: Deploy the Application and SLO Stack

Namespaces and the service

Install the monitoring stack

SLI recording rules and burn-rate alerts

Milestone 5: Run a Chaos Experiment to Validate the SLO Assumptions

Milestone 6: Build the Automated Reliability Pipeline

The SLO gate script

Pushgateway for DORA events

The workflow

Prove the gate works by hand

Milestone 7: Build the DORA Metrics Dashboard

Milestone 8: Write a Blameless Postmortem

Milestone 9: Write an Executable Runbook

Milestone 10: Write an Architecture Decision Record

Validation & Testing

What the finished repository gives you

Quick reference

Common mistakes