Assemble a complete SRE platform - SLO stack with burn-rate alerting, automated reliability pipeline, chaos engineering, DORA metrics, and blameless postmortems in one production-grade repository.
Most DevOps and SRE projects teach one tool in isolation — Prometheus alone, chaos engineering alone, a CI/CD pipeline alone. In a real production team, these pieces are never separate: SLOs drive whether a deployment is allowed to proceed, chaos experiments validate the reliability assumptions those SLOs are built on, and DORA metrics tell you whether all of that automation is actually improving how fast and how safely your team ships. This project assembles everything into one unified SRE platform — the kind of system a senior SRE or platform engineer builds over months at a company like Zerodha or Razorpay, and the kind of thing a hiring manager evaluating a candidate's portfolio actually wants to see: not a list of tools used, but evidence that you understand how those tools work together as one coherent system. ```text The complete platform you are building: +-----------------------------------------------+ | SRE Platform on Kubernetes | | | | +-------------+ +--------------------+ | | | SLO Stack | | Chaos Engineering | | | | Prometheus | | Chaos Mesh + k6 | | | | Grafana | | Experiment Results | | | | Alertmanager| | Dashboard | | | +------+------+ +---------+----------+ | | | | | | +------v------------------------v----------+ | | | Automated Reliability Pipeline | | | | GitHub Actions + SLO gate | | | | Auto-rollback on SLO breach | | | +--------------------+-----------------------+ | | | | | +--------------------v-----------------------+ | | | DORA Metrics Dashboard | | | | Deployment frequency, MTTR, | | | | Change failure rate, Lead time | | | +----------------------------------------------+ | +-----------------------------------------------+ ``` > 💡 **Tip:** Every milestone in this project produces a real, working artifact — a live Kubernetes deployment, an actual firing alert, a real rollback, a written postmortem. By the end you will have a complete GitHub repository you can walk a hiring manager through end to end.
Individually, monitoring, chaos engineering, and CI/CD pipelines each solve a narrow problem. Deployed together but disconnected, they still leave a critical gap: nothing actually *uses* the SLO data to make a decision. A team can have beautiful Grafana dashboards showing an SLO breach in real time and still ship the next release five minutes later, because nothing in the pipeline reads that signal. This project closes that gap deliberately. The SLO stack does not just display data — it feeds a **deployment gate** that automatically halts and reverts a release if the SLO is breached in the minutes after deploy, without waiting for a human to notice the dashboard. Chaos experiments are not run for their own sake — each one tests a specific assumption the SLO error budget policy depends on, and the findings are documented the same way a real incident would be. DORA metrics close the final loop: they tell you objectively whether adding all this automation is actually making the team faster and safer, or just adding process for its own sake. > 📌 **Remember:** A platform like this is judged by whether the pieces talk to each other, not by how many tools appear in the tags. An SLO gate that reads real Prometheus data and triggers a real `kubectl rollout undo` is worth far more, in an interview, than five separate dashboards that never influence a deployment decision.
### Milestone 1: Understand the Core Concepts Before You Build **SLIs, SLOs, and error budgets** are the vocabulary this entire platform is built on. A Service Level Indicator (SLI) is a precise, measurable definition of "good" — for example, "an HTTP response with a status code that is not 500, 502, 503, or 504." A Service Level Objective (SLO) is a target for that indicator over a time window — "99.5% of requests must be good, measured over a rolling 28-day window." The gap between 100% and your SLO target is your **error budget** — the amount of unreliability you are explicitly allowed to spend before you must stop shipping new features and focus entirely on reliability. **Why error budgets change team behaviour, not just dashboards.** The point of defining an error budget is that it converts a vague cultural argument ("should we deploy today or is the system too unstable?") into an objective number anyone on the team can check. An **error budget policy** — a table mapping "percent of budget consumed" to "what the team is allowed to do" — is what turns the SLO from a metric into an actual governance mechanism. **DORA metrics** measure engineering team health at the level above any single service: Deployment Frequency (how often you ship), Lead Time for Changes (how long from commit to production), Change Failure Rate (what percentage of deploys cause a rollback or incident), and MTTR (how fast you recover when something breaks). These four metrics, read together, diagnose your team's actual delivery performance — high deployment frequency with a high change failure rate means you are moving fast and breaking things; low everything means the team is not shipping enough to learn from, regardless of how stable it looks. **Blameless postmortems** exist because "human error" is never a systemic root cause — it's where a shallow investigation stops. A postmortem that ends at "the engineer should have been more careful" has found nothing actionable. A blameless postmortem keeps asking "why" until it reaches a missing test, a missing alert, or a missing piece of automation — something the *system* failed to catch, not something the person failed to remember. ### Milestone 2: Set Up the Project Structure ```bash mkdir sre-platform cd sre-platform ## Create the full directory structure mkdir -p \ kubernetes/namespaces \ kubernetes/app \ kubernetes/monitoring \ kubernetes/chaos \ kubernetes/slos \ pipelines \ postmortems \ runbooks \ docs/adr ## Initialize git git init cat > .gitignore << 'EOF' *.tfstate *.tfstate.backup .terraform/ *.env secrets/ EOF ``` ```text sre-platform/ kubernetes/ namespaces/ <- namespace definitions app/ <- the sample application manifests monitoring/ <- Prometheus, Grafana, Alertmanager config chaos/ <- Chaos Mesh experiment definitions slos/ <- PrometheusRule resources for SLO alerting pipelines/ <- GitHub Actions workflows postmortems/ <- blameless postmortem documents runbooks/ <- executable runbooks docs/ <- SLO spec, DORA definitions, ADRs ``` ### Milestone 3: Define SLOs Before Deploying Anything The first principle of reliability engineering is to define what success looks like *before* you build anything to measure. Writing the SLO document first, with real target numbers and a real error budget policy, forces the design decisions that everything else in this project depends on. ```bash cat > docs/slo-specification.md << 'EOF' # SLO Specification — Orders API Version: 1.0 Date: 2026-08-14 Owner: platform-team ## SLI Definitions ### Availability SLI Good event: HTTP response with status code NOT in {500, 502, 503, 504} Total events: All HTTP requests received by the orders-api service ### Latency SLI Good event: HTTP response delivered within 200ms Total events: All HTTP requests received ## SLO Targets | SLO | Target | Window | Error Budget | | :--- | :--- | :--- | :--- | | Availability | 99.5% | 28 days rolling | 3.6 hours/28 days | | Latency | 95% of requests < 200ms | 28 days rolling | 5% of requests | ## Error Budget Policy | Budget Consumed | Action | | :--- | :--- | | 0% - 50% | Normal development velocity. Deploy freely. | | 50% - 75% | Reduce deployment frequency. Focus on reliability. | | 75% - 100% | Freeze feature deployments. SRE and dev focus on reliability only. | | 100% | Postmortem required. No deploys until root cause resolved. | ## Alert Thresholds (Multi-Window Burn Rate) | Alert | Window | Burn Rate | Action | | :--- | :--- | :--- | :--- | | Critical | 1h + 5m | 14.4x | Page on-call immediately | | High | 6h + 1h | 6x | Slack to team channel | | Medium | 3d + 6h | 1x | Ticket for next sprint | EOF ``` > 📌 **Remember:** Writing the error budget policy table is not optional bureaucracy — it's the piece that makes the SLO actually govern deployment decisions in Milestone 6, instead of sitting in a dashboard nobody consults before shipping. ### Milestone 4: Deploy the Application and SLO Stack ```bash ## Namespaces cat > kubernetes/namespaces/namespaces.yaml << 'EOF' apiVersion: v1 kind: Namespace metadata: name: production labels: environment: production --- apiVersion: v1 kind: Namespace metadata: name: monitoring labels: purpose: observability --- apiVersion: v1 kind: Namespace metadata: name: chaos-testing labels: purpose: chaos-engineering EOF kubectl apply -f kubernetes/namespaces/namespaces.yaml ``` Deploy the application that every other milestone in this project targets: ```bash cat > kubernetes/app/deployment.yaml << 'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: orders-api namespace: production labels: app: orders-api version: v1 spec: replicas: 3 selector: matchLabels: app: orders-api template: metadata: labels: app: orders-api version: v1 annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" prometheus.io/path: "/metrics" spec: containers: - name: api image: nginx:alpine ports: - containerPort: 80 resources: requests: memory: "64Mi" cpu: "100m" limits: memory: "128Mi" cpu: "200m" readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 10 failureThreshold: 3 livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 15 periodSeconds: 20 --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: orders-api-pdb namespace: production spec: minAvailable: 2 selector: matchLabels: app: orders-api --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: orders-api-hpa namespace: production spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: orders-api minReplicas: 3 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 EOF kubectl apply -f kubernetes/app/deployment.yaml ``` > 🔴 **Common Mistake:** Deploying a bare Deployment with no PodDisruptionBudget or readiness probe means the reliability pipeline in Milestone 6 has nothing to actually protect — a rollback triggered by an SLO breach only helps if the underlying deployment has the primitives (PDB, HPA, readiness gates) to recover gracefully once traffic shifts back. Install the monitoring stack: ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install kube-prometheus-stack \ prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --set grafana.adminPassword="SREPlatform2026!" \ --wait ``` Apply the SLO burn-rate alert rules: ```bash cat > kubernetes/slos/orders-api-slos.yaml << 'EOF' apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: orders-api-slo-rules namespace: monitoring labels: release: kube-prometheus-stack spec: groups: - name: orders-api-slo-recording interval: 30s rules: - record: orders_api:availability:rate5m expr: | 1 - rate(nginx_http_requests_total{status=~"5.."}[5m]) / rate(nginx_http_requests_total[5m]) - record: orders_api:availability:rate1h expr: | 1 - rate(nginx_http_requests_total{status=~"5.."}[1h]) / rate(nginx_http_requests_total[1h]) - name: orders-api-slo-alerts rules: - alert: OrdersAPICriticalBurnRate expr: | (1 - orders_api:availability:rate5m) > (14.4 * 0.005) and (1 - orders_api:availability:rate1h) > (14.4 * 0.005) for: 2m labels: severity: critical slo: orders-api-availability annotations: summary: "Orders API burning error budget at 14x rate" description: > Availability is {{ $value | humanizePercentage }}. Error budget will be exhausted in approximately 2 hours. EOF kubectl apply -f kubernetes/slos/orders-api-slos.yaml ``` ### Milestone 5: Run a Chaos Experiment to Validate the SLO Assumptions Before trusting the SLO gate in the next milestone, confirm the reliability primitives you deployed (PDB, readiness probes) actually protect the SLO under real failure — this is the same experiment cycle from the standalone chaos engineering project, applied here to validate this specific platform's assumptions. ```bash cat > kubernetes/chaos/pod-kill-validation.yaml << 'EOF' apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: sre-platform-pod-kill namespace: chaos-testing spec: action: pod-kill selector: namespaces: - production labelSelectors: app: orders-api mode: one duration: "2m" EOF ## Hypothesis: with the PDB (minAvailable: 2) and readiness probe ## already in place, killing one of three pods should NOT breach ## the 99.5% availability SLO defined in docs/slo-specification.md kubectl apply -f kubernetes/chaos/pod-kill-validation.yaml kubectl get pods -n production --watch ``` ```text NAME READY STATUS orders-api-7d9f8c6b-9rvbf 1/1 Running orders-api-7d9f8c6b-9rvbf 0/1 Terminating orders-api-7d9f8c6b-9rvbf 0/1 Pending orders-api-7d9f8c6b-9rvbf 0/1 Running orders-api-7d9f8c6b-9rvbf 1/1 Running ``` ```bash kubectl delete -f kubernetes/chaos/pod-kill-validation.yaml ``` > 💡 **Tip:** Check the `orders_api:availability:rate5m` recording rule in Prometheus immediately after this experiment. If it dips below 99.5% even briefly, that's real evidence your PDB and readiness probe configuration needs tuning *before* you rely on the automated SLO gate in Milestone 6 — validate the safety net before you start using it to make automated decisions. ### Milestone 6: Build the Automated Reliability Pipeline This is the piece that makes the platform genuinely automated rather than just observable: a GitHub Actions pipeline that deploys, waits for metrics to settle, checks the SLO, and automatically rolls back if the deployment breached it — without a human needing to notice the dashboard first. ```bash cat > pipelines/deploy-with-slo-gate.yml << 'EOF' name: Deploy with SLO Gate on: push: branches: [main] env: IMAGE_TAG: ${{ github.sha }} jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Configure kubectl uses: azure/k8s-set-context@v3 with: kubeconfig: ${{ secrets.KUBECONFIG }} - name: Deploy new version run: | kubectl set image deployment/orders-api \ api=nginx:${{ github.sha }} \ -n production ## Wait for rollout to complete before checking SLO kubectl rollout status deployment/orders-api \ -n production \ --timeout=300s ## Wait for metrics to settle before evaluating the SLO — ## checking immediately after rollout catches deploy-time ## noise, not the deployment's real steady-state behaviour - name: Wait for metrics to settle run: sleep 300 - name: Check SLO compliance id: slo-check run: | ## Query Prometheus for the current error rate ## Replace with your actual in-cluster Prometheus URL ERROR_RATE=$(curl -s \ "http://prometheus:9090/api/v1/query?query=1-orders_api:availability:rate5m" \ | python3 -c " import sys, json data = json.load(sys.stdin) value = float(data['data']['result'][0]['value'][1]) print(value) ") echo "Current error rate: $ERROR_RATE" echo "error_rate=$ERROR_RATE" >> $GITHUB_OUTPUT ## Fail this step if error rate exceeds the SLO threshold python3 -c " rate = float('$ERROR_RATE') threshold = 0.005 if rate > threshold: print(f'ERROR: Error rate {rate:.4f} exceeds SLO threshold {threshold}') exit(1) else: print(f'OK: Error rate {rate:.4f} within SLO') " ## Automatic rollback fires only if the SLO check above failed - name: Rollback on SLO breach if: failure() && steps.slo-check.outcome == 'failure' run: | echo "SLO breach detected — rolling back" kubectl rollout undo deployment/orders-api \ -n production kubectl rollout status deployment/orders-api \ -n production \ --timeout=300s ## Notify the team with full context, not just "it failed" curl -X POST ${{ secrets.SLACK_WEBHOOK }} \ -H 'Content-type: application/json' \ -d '{ "text": ":rotating_light: *AUTO-ROLLBACK* orders-api rolled back after SLO breach", "attachments": [{ "color": "danger", "text": "Commit: ${{ github.sha }}\nError rate exceeded 0.5% SLO threshold" }] }' EOF ``` > 🔴 **Common Mistake:** Checking the SLO immediately after `kubectl rollout status` returns, with no settle time, means you're measuring deploy-time noise (connection resets during the rolling update itself) rather than the new version's real steady-state error rate. The `sleep 300` step is deliberate — five minutes is enough for the rollout's own transient errors to clear before the SLO gate makes its decision. ### Milestone 7: Build the DORA Metrics Dashboard DORA metrics answer a different question than the SLO stack does: not "is the service healthy right now," but "is our engineering process actually getting faster and safer over time." ```bash cat > docs/dora-metrics-definition.md << 'EOF' # DORA Metrics — Measurement Definitions ## Deployment Frequency Definition: How often we deploy to production Measurement: Count of successful GitHub Actions deploy jobs per week Target: Daily (Elite) PromQL: increase(github_actions_job_total{job_name="deploy", result="success"}[7d]) ## Lead Time for Changes Definition: Time from commit merge to code running in production Measurement: Time between push event and successful deploy job completion Target: Less than 1 day (Elite) ## Change Failure Rate Definition: Percentage of deploys that cause a rollback or incident Measurement: Rollback events / total deploy events Target: Less than 5% (Elite) PromQL: rate(deploy_rollbacks_total[30d]) / rate(deploy_total[30d]) ## MTTR — Mean Time to Restore Definition: How long it takes to restore service after an incident Measurement: Time from incident alert firing to alert resolving Target: Less than 1 hour (Elite) EOF ``` Build the Grafana dashboard JSON directly, wiring each panel to the metrics defined above and to the SLO recording rules from Milestone 4: ```bash cat > kubernetes/monitoring/dora-dashboard.json << 'EOF' { "title": "DORA Metrics — Platform Health", "panels": [ { "title": "Deployment Frequency (per week)", "type": "stat", "targets": [{ "expr": "increase(kube_deployment_status_observed_generation{deployment='orders-api'}[7d])" }], "gridPos": {"x": 0, "y": 0, "w": 6, "h": 4} }, { "title": "Error Budget Remaining", "type": "gauge", "targets": [{ "expr": "(orders_api:availability:rate28d - 0.995) / 0.005 * 100" }], "gridPos": {"x": 6, "y": 0, "w": 6, "h": 4}, "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100, "thresholds": { "steps": [ {"color": "red", "value": 0}, {"color": "yellow", "value": 25}, {"color": "green", "value": 50} ] } } } }, { "title": "Rollback Events (30 days)", "type": "stat", "targets": [{ "expr": "increase(deploy_rollbacks_total[30d])" }], "gridPos": {"x": 12, "y": 0, "w": 6, "h": 4} }, { "title": "SLO Compliance Over Time", "type": "timeseries", "targets": [{ "expr": "orders_api:availability:rate1h * 100", "legendFormat": "Availability %" }], "gridPos": {"x": 0, "y": 4, "w": 24, "h": 8} } ] } EOF ``` > 📌 **Remember:** Deployment frequency high *and* change failure rate low, together, is the actual target — not either metric alone. High deployment frequency with a high change failure rate means you're shipping fast and breaking things; low everything means the team isn't deploying often enough to learn anything meaningful from these numbers. ### Milestone 8: Write a Blameless Postmortem A platform like this needs a postmortem culture as much as it needs dashboards — the following is a complete, realistic example demonstrating what a genuinely blameless, systemically-rooted postmortem looks like. ```bash cat > postmortems/2026-08-14-orders-api-latency-incident.md << 'EOF' # Postmortem — Orders API Latency Spike Date: 2026-08-14 Severity: P2 Duration: 47 minutes Authors: Arjun Sharma ## Summary The orders-api experienced elevated p99 latency (450ms vs 200ms SLO) from 14:32 to 15:19 IST on 2026-08-14. Affected approximately 15,000 users during peak lunch ordering hours. Root cause: deployment of v2.1.3 with an unindexed database query that caused table scans under load. ## Impact * Duration: 47 minutes * Users affected: ~15,000 (estimated from traffic x error rate) * Error budget consumed: 18% of monthly budget in one incident ## Timeline 14:28 IST — v2.1.3 deployed via automated pipeline 14:32 IST — Grafana alert fires: p99 latency above 200ms 14:35 IST — On-call Arjun acknowledges alert 14:38 IST — Arjun checks deployment timeline, correlates with 14:28 deploy 14:45 IST — Arjun checks database slow query log, finds full table scans 14:52 IST — Confirms unindexed query introduced in v2.1.3 migration 14:55 IST — Arjun initiates rollback (WRONG SERVICE — targeted the similarly-named orders-worker instead of orders-api) 15:02 IST — Latency unchanged — realises rollback targeted wrong deployment 15:05 IST — Rolls back the correct deployment (orders-api) 15:14 IST — Latency returns to normal (p99: 45ms) 15:19 IST — Alert resolves — MTTR: 47 minutes ## Root Cause Database migration in v2.1.3 added a new column to the orders table without an index. A new query pattern added in the same release performs a WHERE clause on this unindexed column. Under load, every query performs a full table scan on the 45M-row orders table. ## Contributing Factors 1. No database query performance test in CI pipeline 2. Load test uses 1,000 rows of test data — misses slow-query behaviour 3. On-call rolled back the wrong service — similar service names caused confusion 4. No staging environment with production-scale data volume ## Five Whys Why did users experience slow responses? Database queries were performing full table scans Why were queries doing full table scans? The WHERE column had no index Why was the index missing? The migration script added the column but not the index Why wasn't this caught before production? No slow query analysis exists in the CI pipeline Why is there no slow query analysis in CI? Nobody has built it — this is the systemic root cause ## Action Items | Action | Owner | Due | Status | | :--- | :--- | :--- | :--- | | Add EXPLAIN ANALYZE to CI for all new queries | Dev team | 2026-08-21 | Open | | Seed staging database with production data shape | Arjun | 2026-08-28 | Open | | Add index on orders.status column | Database team | 2026-08-16 | Closed | | Rename orders-worker to orders-processor to prevent confusion | Arjun | 2026-08-19 | Open | | Add rollback target verification step to pipeline | Arjun | 2026-08-21 | Open | ## Lessons Learned * Always verify the target service name before executing a rollback — a simple confirmation step would have saved roughly 10 minutes * Load tests using unrealistically small datasets miss exactly the performance issues that matter most in production * The systemic fix is automated slow-query detection in CI — not asking the on-call engineer to be more careful next time ## SLO Impact Error budget consumed: 18% (8.2 hours of the allowed 47.5 hours) Remaining budget for August: 82% EOF ``` > 🔴 **Common Mistake:** Ending the Five Whys at "the on-call engineer rolled back the wrong service" instead of continuing to "why did two services have confusingly similar names with no rollback confirmation step" stops at human error instead of reaching a systemic, fixable cause. Every postmortem in this project's `postmortems/` folder should end at a missing test, alert, or automation — never at a person. ### Milestone 9: Write an Executable Runbook Runbooks turn tribal knowledge — "when this alert fires, I usually check X, then Y" — into something anyone on the team can run, including someone paged for the first time on this service. ```bash cat > runbooks/orders-api-high-error-rate.py << 'EOF' #!/usr/bin/env python3 """ Runbook: Orders API High Error Rate Use this when the OrdersAPICriticalBurnRate alert fires. This runbook provides a systematic investigation path. Usage: python3 runbooks/orders-api-high-error-rate.py """ import subprocess from datetime import datetime def run(cmd): """Run a shell command and return output.""" result = subprocess.run(cmd, shell=True, capture_output=True, text=True) return result.stdout.strip() def check_pod_status(): print("\n[1/5] Checking pod status...") output = run("kubectl get pods -n production -l app=orders-api") print(output) if "CrashLoopBackOff" in output or "Error" in output: print(" FINDING: Pods in error state. Check pod logs.") return False print(" OK: All pods appear healthy") return True def check_recent_deployments(): print("\n[2/5] Checking recent deployments...") output = run("kubectl rollout history deployment/orders-api -n production") print(output) print(" ACTION: Was there a recent deployment? Check with git log.") def check_pod_logs(): print("\n[3/5] Checking pod logs for errors...") output = run( "kubectl logs -n production -l app=orders-api " "--tail=50 --prefix=true | grep -i error | head -20" ) if output: print("ERRORS FOUND:") print(output) else: print(" No ERROR lines in recent logs") def check_resource_usage(): print("\n[4/5] Checking resource usage...") output = run("kubectl top pods -n production -l app=orders-api") print(output) print(" Check if any pod is near its memory or CPU limit") def rollback_prompt(): print("\n[5/5] Rollback decision...") recent = run("kubectl rollout history deployment/orders-api -n production") print(recent) answer = input("\nRollback the deployment? (yes/no): ").strip().lower() if answer == "yes": print("Rolling back...") run("kubectl rollout undo deployment/orders-api -n production") print("Rollback complete. Monitor Grafana for recovery.") else: print("Rollback skipped. Continue investigation.") if __name__ == "__main__": print("Orders API High Error Rate Runbook") print(f"Started: {datetime.now().isoformat()}") print("=" * 50) check_pod_status() check_recent_deployments() check_pod_logs() check_resource_usage() rollback_prompt() print(f"\nCompleted: {datetime.now().isoformat()}") print("Remember to write a postmortem if this was a P1 or P2 incident.") EOF chmod +x runbooks/orders-api-high-error-rate.py ``` ### Milestone 10: Write an Architecture Decision Record An ADR documents *why* a significant technical decision was made — not just what was chosen — so future engineers (including you, six months later) understand the trade-offs without re-litigating the decision. ```bash cat > docs/adr/ADR-001-why-prometheus-not-datadog.md << 'EOF' # ADR 001 — Prometheus over Datadog for Metrics Date: 2026-08-14 Status: Accepted Deciders: Arjun Sharma, Platform Team ## Context We need a metrics collection and alerting system for our Kubernetes cluster. Two options were evaluated: Prometheus (open-source, self-hosted) and Datadog (SaaS). ## Decision We chose Prometheus with Grafana. ## Reasons * Cost: Datadog costs $15-25 per host per month. With 20 nodes, that is roughly $3,600-6,000 per year. Prometheus is free to run, though not free to operate. * Control: Prometheus runs inside our own cluster. Datadog requires sending all metrics to Datadog's cloud, which raises a data residency concern for financial transaction data. * SLO alerting: Prometheus and PrometheusRule resources integrate natively with Kubernetes and allow alert rules to be managed as code in Git, reviewed the same way application code is. ## Trade-offs * Datadog has better out-of-the-box dashboards and APM tracing * Prometheus requires more in-house operational expertise to run reliably * Datadog has stronger built-in anomaly detection features ## Consequences * We will invest engineering time in building our own Grafana dashboards * Alert rules are managed as PrometheusRule Kubernetes resources in Git * Long-term metric retention beyond 15 days will require Thanos or Cortex ## Review Date 2027-08-14 — revisit this decision if the engineering team grows beyond 50 people, at which point operational overhead may outweigh the cost savings. EOF ```
When every milestone above is complete, you have a single GitHub repository containing: * Live Kubernetes deployments with PodDisruptionBudget, HPA, and readiness probes * A Prometheus stack with multi-window burn-rate SLO alerting * An automated deployment pipeline with a real SLO gate and automatic rollback * A Chaos Mesh experiment that validates the SLO's own reliability assumptions * A DORA metrics dashboard showing deployment frequency and MTTR * A blameless postmortem demonstrating systemic root-cause analysis * An executable Python runbook for the platform's primary alert * An Architecture Decision Record explaining why each major tool was chosen This is the artifact that demonstrates senior SRE thinking — not just using individual tools, but showing why and how they connect into one system.
```bash # 1. Confirm the application is deployed with all reliability primitives kubectl get deployment,pdb,hpa -n production # Expected: orders-api Deployment, orders-api-pdb PDB, orders-api-hpa HPA # 2. Confirm the SLO PrometheusRule is loaded kubectl get prometheusrule -n monitoring # Expected: orders-api-slo-rules is listed # 3. Confirm the chaos experiment does not breach the SLO kubectl apply -f kubernetes/chaos/pod-kill-validation.yaml # Watch Grafana — orders_api:availability:rate5m should stay above 99.5% kubectl delete -f kubernetes/chaos/pod-kill-validation.yaml # 4. Confirm the pipeline's SLO gate step is syntactically valid cat pipelines/deploy-with-slo-gate.yml | python3 -c \ "import sys, yaml; yaml.safe_load(sys.stdin)" # Expected: no output means valid YAML # 5. Confirm the runbook executes without errors python3 runbooks/orders-api-high-error-rate.py # Expected: walks through all 5 diagnostic steps and prompts for rollback # 6. Confirm the postmortem and ADR exist and are committed git log --oneline -- postmortems/ docs/adr/ ```
| Component | Access | Purpose | | :--- | :--- | :--- | | Grafana | `kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring` | SLO and DORA dashboards | | Prometheus | `kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring` | Query raw metrics | | Chaos Dashboard | `kubectl port-forward svc/chaos-dashboard 2333:2333 -n chaos-mesh` | Manage chaos experiments | | Runbook | `python3 runbooks/orders-api-high-error-rate.py` | Guided incident response |
Most DevOps and SRE projects teach one tool in isolation — Prometheus alone, chaos engineering alone, a CI/CD pipeline a...
Individually, monitoring, chaos engineering, and CI/CD pipelines each solve a narrow problem. Deployed together but disc...
Milestone 1: Understand the Core Concepts Before You Build SLIs, SLOs, and error budgets are the vocabulary this entire ...
When every milestone above is complete, you have a single GitHub repository containing: Live Kubernetes deployments with...
...
Component Access Purpose Grafana kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring SLO and DO...
Building the platform without an application to run it against produces dashboards with no data. Deploy a real, multi-re...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.