A Kubernetes cluster with no monitoring is flying blind. You cannot tell if your application is slow, if pods are crashing, or if a node is about to run out of memory — until users complain or something breaks outright. This project builds the full observability stack used by SRE teams in real production environments: Prometheus collects metrics, Grafana visualizes them, and Alertmanager sends a Slack message when something goes wrong. Beyond basic monitoring, you will define real SLOs (Service Level Objectives) and write burn-rate alerts that fire *before* you breach them — not after. ```text Kubernetes cluster | kube-prometheus-stack (Helm chart) |-- Prometheus (collects and stores metrics) |-- Grafana (visualizes metrics as dashboards) |-- Alertmanager (routes alerts to Slack) |-- Node Exporter (CPU, memory, disk metrics per node) |-- kube-state-metrics (pod, deployment, service state) ``` > 💡 **Tip:** By the end of this project you will understand the difference between monitoring that generates noise (alerts nobody trusts) and monitoring that generates signal (alerts that mean something is actually wrong).
Without monitoring, the only signal that something is broken is a user complaint — which means the problem has already been affecting real people for an unknown amount of time. Imagine a streaming platform like Hotstar during a high-traffic cricket match: basic threshold alerting ("alert when error rate is above 1%") is only marginally better, since it fires constantly for brief, harmless spikes and can miss sustained problems sitting just below the threshold. This project solves both problems. First, it gives you full visibility into cluster and application health through Prometheus and Grafana. Second, it replaces naive threshold alerting with **burn rate alerting**, which asks a much better question: *how fast are we consuming our error budget?* A service burning its budget 14x faster than normal will exhaust it in about 2 hours — that deserves an immediate page. A service burning at 1x normal rate will exhaust it at the end of the month — that deserves a ticket, not a 3am wake-up call. > 📌 **Remember:** An SLO of "99% availability over 28 days" gives you an error budget — the 1% of requests allowed to fail. If you've used half that budget by day 14, something is wrong and needs investigating before the budget runs out completely.
### Milestone 1: Understand the Core Concepts Before You Build **Prometheus** is a metrics collection and storage system. Unlike systems where applications push data outward, Prometheus *pulls* (scrapes) metrics from your applications on a regular interval — typically every 15–30 seconds — by visiting a `/metrics` HTTP endpoint that each component exposes. ```text Every 30 seconds: Prometheus -> HTTP GET /metrics -> target -> parse -> store ``` Metrics are stored as a name, a set of labels, and a value at a point in time, for example: ```text http_requests_total{method="GET", status="200", service="orders"} 1523 ``` This one line means: 1523 GET requests to the `orders` service returned HTTP 200. An **SLO (Service Level Objective)** is a target for how reliable a service must be, expressed as a percentage over a time window — for example, "99% of requests must succeed over a rolling 28-day window." The allowed failure percentage is your **error budget**. **Burn rate alerting** measures how fast you're consuming that error budget, rather than just checking if you've crossed a fixed threshold. This catches both fast, severe outages and slow, sustained degradation — with different urgency for each. ### Milestone 2: Set Up a Local Kubernetes Cluster Kind (Kubernetes in Docker) gives you a real, multi-node cluster on your own laptop. ```bash # Install Kind brew install kind # or on Linux: # curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-amd64 # chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind # Install kubectl curl -LO "https://dl.k8s.io/release/$(curl -Ls \ https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl sudo mv kubectl /usr/local/bin/ # Install Helm curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ``` Create a 3-node cluster: ```bash cat > kind-cluster.yaml << 'EOF' kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker EOF kind create cluster \ --name monitoring-project \ --config kind-cluster.yaml kubectl get nodes ``` ```text NAME STATUS ROLES monitoring-project-control-plane Ready control-plane monitoring-project-worker Ready <none> monitoring-project-worker2 Ready <none> ``` ### Milestone 3: Deploy the kube-prometheus-stack This single Helm chart installs Prometheus, Grafana, Alertmanager, Node Exporter, and kube-state-metrics, all pre-configured to work together. ```bash helm repo add prometheus-community \ https://prometheus-community.github.io/helm-charts helm repo update kubectl create namespace monitoring ``` ```bash cat > monitoring-values.yaml << 'EOF' grafana: adminPassword: "ChangeThisPassword2026!" ingress: enabled: false prometheus: prometheusSpec: retention: 15d storageSpec: volumeClaimTemplate: spec: resources: requests: storage: 10Gi alertmanager: alertmanagerSpec: storage: volumeClaimTemplate: spec: resources: requests: storage: 1Gi EOF helm install kube-prometheus-stack \ prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --values monitoring-values.yaml \ --wait ``` ```bash kubectl get pods -n monitoring ``` ```text NAME READY STATUS alertmanager-kube-prometheus-stack-alertmanager-0 2/2 Running kube-prometheus-stack-grafana-7b9b4c8dc-xkq2p 3/3 Running kube-prometheus-stack-operator-6c7f89c5b4-wl9vp 1/1 Running prometheus-kube-prometheus-stack-prometheus-0 2/2 Running ``` > ⚠️ **Security:** The Grafana admin password shown here is for local learning only. Never use a hardcoded plaintext password like this in a real cluster — use a Kubernetes Secret or your secrets manager instead. ### Milestone 4: Deploy a Sample Application to Monitor ```bash cat > sample-app.yaml << 'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: orders-api namespace: default labels: app: orders-api spec: replicas: 3 selector: matchLabels: app: orders-api template: metadata: labels: app: orders-api annotations: prometheus.io/scrape: "true" prometheus.io/port: "80" prometheus.io/path: "/metrics" spec: containers: - name: orders-api image: nginx:alpine ports: - containerPort: 80 resources: requests: memory: "64Mi" cpu: "100m" limits: memory: "128Mi" cpu: "200m" --- apiVersion: v1 kind: Service metadata: name: orders-api namespace: default spec: selector: app: orders-api ports: - port: 80 targetPort: 80 EOF kubectl apply -f sample-app.yaml kubectl get pods -l app=orders-api ``` ### Milestone 5: Access Grafana and Import Dashboards ```bash kubectl port-forward -n monitoring \ svc/kube-prometheus-stack-grafana 3000:80 ``` Open `http://localhost:3000` and log in with username `admin` and the password you set in Milestone 3. Import two starter dashboards: 1. Click `+` → Import → enter dashboard ID `1860` (Node Exporter Full) → select Prometheus as the data source → Import. 2. Repeat with dashboard ID `15661` (Kubernetes Cluster Overview). You now have real CPU, memory, disk, and cluster-wide dashboards without building anything from scratch. ### Milestone 6: Write SLO Alert Rules with Burn Rate Logic First, explore the raw metrics in Prometheus: ```bash kubectl port-forward -n monitoring \ svc/kube-prometheus-stack-prometheus 9090:9090 ``` Open `http://localhost:9090` and try these queries: ```text # Total request rate over the last 5 minutes rate(nginx_http_requests_total[5m]) # Error rate — requests returning 5xx rate(nginx_http_requests_total{status=~"5.."}[5m]) ``` > 📌 **Remember:** `rate()` calculates the per-second rate of change of a counter. `[5m]` means "look at the last 5 minutes." `status=~"5.."` is a regex matching any status code starting with 5 (500, 502, 503, 504). Now define recording rules (pre-computed queries) and burn-rate alerts: ```bash cat > slo-alert-rules.yaml << 'EOF' apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: orders-api-slo-rules namespace: monitoring labels: release: kube-prometheus-stack spec: groups: - name: orders-api-slo-recording interval: 30s rules: - record: orders_api:error_ratio:rate5m expr: | rate(nginx_http_requests_total{status=~"5.."}[5m]) / rate(nginx_http_requests_total[5m]) - record: orders_api:error_ratio:rate1h expr: | rate(nginx_http_requests_total{status=~"5.."}[1h]) / rate(nginx_http_requests_total[1h]) - record: orders_api:error_ratio:rate6h expr: | rate(nginx_http_requests_total{status=~"5.."}[6h]) / rate(nginx_http_requests_total[6h]) - record: orders_api:error_ratio:rate1d expr: | rate(nginx_http_requests_total{status=~"5.."}[1d]) / rate(nginx_http_requests_total[1d]) # SLO target: 99% availability over 28 days # Error budget: 1% = ~403 minutes of downtime per 28 days - name: orders-api-slo-alerts rules: - alert: OrdersAPIHighBurnRate expr: | orders_api:error_ratio:rate5m > (14.4 * 0.01) and orders_api:error_ratio:rate1h > (14.4 * 0.01) for: 2m labels: severity: critical slo: "orders-api-availability" annotations: summary: "Orders API burning error budget at 14x rate" description: > Error ratio is {{ $value | humanizePercentage }}. At this rate the 28-day error budget is exhausted in roughly 2 hours. Immediate investigation required. - alert: OrdersAPIMediumBurnRate expr: | orders_api:error_ratio:rate1h > (6 * 0.01) and orders_api:error_ratio:rate6h > (6 * 0.01) for: 15m labels: severity: warning slo: "orders-api-availability" annotations: summary: "Orders API burning error budget at 6x rate" description: > Error ratio is {{ $value | humanizePercentage }}. At this rate the budget is exhausted in roughly 5 days. - alert: OrdersAPIPodCrashLooping expr: | increase(kube_pod_container_status_restarts_total{ namespace="default", pod=~"orders-api-.*" }[30m]) > 3 labels: severity: critical annotations: summary: "Orders API pod is crash looping" description: > Pod {{ $labels.pod }} restarted more than 3 times in the last 30 minutes. - alert: OrdersAPIHighMemoryUsage expr: | container_memory_working_set_bytes{ namespace="default", pod=~"orders-api-.*" } / kube_pod_container_resource_limits{ namespace="default", resource="memory", pod=~"orders-api-.*" } > 0.85 for: 5m labels: severity: warning annotations: summary: "Orders API memory usage above 85%" description: > Pod {{ $labels.pod }} is using {{ $value | humanizePercentage }} of its memory limit. EOF kubectl apply -f slo-alert-rules.yaml ``` > 🔴 **Common Mistake:** Writing burn-rate alerts directly against raw metrics instead of recording rules forces Prometheus to scan millions of raw data points on every evaluation. Recording rules pre-compute the 5m/1h/6h/1d ratios once, on a schedule, so alert evaluation stays fast. This is not optional at any real scale. ### Milestone 7: Configure Slack Alerts 1. Go to `api.slack.com/apps` → Create New App → From scratch. 2. Name it `Monitoring-Alerts` and select your workspace. 3. Click Incoming Webhooks → turn it on → Add New Webhook to Workspace. 4. Choose or create a channel (e.g. `#alerts-dev`) → Allow. 5. Copy the webhook URL — it looks like `https://hooks.slack.com/services/T.../B.../xxx`. ```bash cat > alertmanager-config.yaml << 'EOF' alertmanager: config: global: resolve_timeout: 5m route: receiver: slack-alerts group_by: [alertname, severity] group_wait: 30s group_interval: 5m repeat_interval: 4h receivers: - name: slack-alerts slack_configs: - api_url: "YOUR_SLACK_WEBHOOK_URL_HERE" send_resolved: true channel: "#alerts-dev" title: | [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} text: | {{ range .Alerts }} *Alert:* {{ .Labels.alertname }} *Severity:* {{ .Labels.severity }} *Summary:* {{ .Annotations.summary }} {{ end }} EOF helm upgrade kube-prometheus-stack \ prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --values monitoring-values.yaml \ --values alertmanager-config.yaml ``` > 🔴 **Common Mistake:** Routing every severity into one Slack channel causes alert fatigue — engineers stop reading a channel that pages them at the same volume for both critical outages and minor warnings. Route `severity: critical` to a dedicated on-call channel, and `severity: warning` to a lower-priority channel reviewed during business hours. ### Milestone 8: Build an SLO Dashboard in Grafana In Grafana, go to Dashboards → New Dashboard, and add these four panels — each is one PromQL query pointed at your recording rules from Milestone 6. | Panel | Query | | :--- | :--- | | Current Error Rate | `orders_api:error_ratio:rate5m * 100` | | Error Budget Remaining | `(1 - orders_api:error_ratio:rate1d) / 0.01 * 100` | | Requests per Second | `sum(rate(nginx_http_requests_total[5m]))` | Save the dashboard as `Orders API — SLO Overview`. > 💡 **Tip:** Export every hand-built dashboard as JSON (Dashboard Settings → JSON Model) and commit it to Git. Grafana stores dashboards inside its own pod storage — if that pod is deleted and recreated, unsaved dashboards are gone permanently.
Confirm the full stack is working end to end: ```bash # 1. Confirm all monitoring pods are Running kubectl get pods -n monitoring # Expected: every pod shows STATUS Running # 2. Confirm Prometheus loaded the custom rules kubectl get prometheusrule -n monitoring # Expected: orders-api-slo-rules is listed # 3. Confirm Grafana is reachable and shows live data kubectl port-forward -n monitoring \ svc/kube-prometheus-stack-grafana 3000:80 # Visit http://localhost:3000 — dashboards should show real metrics # 4. Send a test alert to confirm the Slack webhook works curl -H 'Content-type: application/json' \ --data '{"text":"Test alert from Alertmanager setup"}' \ YOUR_SLACK_WEBHOOK_URL_HERE # Expected: message appears in your #alerts-dev channel ``` If everything above passes, your SLO alert rules will show as `PENDING` or `FIRING` (not `INACTIVE`) in the Prometheus Alerts UI once error rates actually cross the burn-rate thresholds.
Installing kube-prometheus-stack without setting resource limits will exhaust memory on a small local cluster. Prometheus holds metrics in memory before flushing to disk, and on a 2-core, 4GB Kind cluster the default configuration can consume all available memory within an hour. Always set explicit resource limits in your values file for local testing. Writing burn-rate alert expressions directly against raw counters instead of recording rules forces Prometheus to scan enormous amounts of raw data on every evaluation cycle. Recording rules are required, not optional, once you move past a toy example. Setting an SLO target of 99.999% without the engineering maturity to support it creates an error budget so small it's unusable — 99.999% only allows about 26 seconds of downtime per month, and any routine deployment can consume a meaningful chunk of that. Start at 99% or 99.5% and tighten the target only as reliability actually improves. Routing all alerts, regardless of severity, into a single Slack channel trains engineers to ignore that channel entirely. Split critical (page-worthy) alerts from warning (next-business-day) alerts into separate channels from the start. Not exporting Grafana dashboards as JSON means any manually built dashboard is lost the moment the Grafana pod restarts, since dashboards live in the pod's own storage by default.
A Kubernetes cluster with no monitoring is flying blind. You cannot tell if your application is slow, if pods are crashi...
Without monitoring, the only signal that something is broken is a user complaint — which means the problem has already b...
Milestone 1: Understand the Core Concepts Before You Build Prometheus is a metrics collection and storage system. Unlike...
Confirm the full stack is working end to end: If everything above passes, your SLO alert rules will show as PENDING or F...
Installing kube-prometheus-stack without setting resource limits will exhaust memory on a small local cluster. Prometheu...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.