Deploy Kubernetes Monitoring with Prometheus, Grafana, and SLO Alerting

Without monitoring, the only signal that something is broken is a user complaint — which means the problem has already been affecting real people for an unknown amount of time. Imagine a streaming platform like Hotstar during a high-traffic cricket match: basic threshold alerting ("alert when error rate is above 1%") is only marginally better, since it fires constantly for brief, harmless spikes and can miss sustained problems sitting just below the threshold.

This project solves both problems. First, it gives you full visibility into cluster and application health through Prometheus and Grafana. Second, it replaces naive threshold alerting with burn rate alerting, which asks a much better question: how fast are we consuming our error budget? A service burning its budget 14x faster than normal will exhaust it in about 2 hours — that deserves an immediate page. A service burning at 1x normal rate will exhaust it at the end of the month — that deserves a ticket, not a 3am wake-up call.

Remember

An SLO of "99% availability over 28 days" gives you an error budget — the 1% of requests allowed to fail. If you've used half that budget by day 14, something is wrong and needs investigating before the budget runs out completely.

Milestone 1: Understand the Core Concepts Before You Build

Prometheus is a metrics collection and storage system. Unlike systems where applications push data outward, Prometheus pulls (scrapes) metrics from your applications on a regular interval — typically every 15–30 seconds — by visiting a /metrics HTTP endpoint that each component exposes.

Metrics are stored as a name, a set of labels, and a value at a point in time, for example:

This one line means: 1523 GET requests to the orders service returned HTTP 200.

An SLO (Service Level Objective) is a target for how reliable a service must be, expressed as a percentage over a time window — for example, "99% of requests must succeed over a rolling 28-day window." The allowed failure percentage is your error budget.

Burn rate alerting measures how fast you're consuming that error budget, rather than just checking if you've crossed a fixed threshold. This catches both fast, severe outages and slow, sustained degradation — with different urgency for each.

Milestone 2: Set Up a Local Kubernetes Cluster

Kind (Kubernetes in Docker) gives you a real, multi-node cluster on your own laptop.

Create a 3-node cluster:

Milestone 3: Deploy the kube-prometheus-stack

This single Helm chart installs Prometheus, Grafana, Alertmanager, Node Exporter, and kube-state-metrics, all pre-configured to work together.

Security

The Grafana admin password shown here is for local learning only. Never use a hardcoded plaintext password like this in a real cluster — use a Kubernetes Secret or your secrets manager instead.

Milestone 4: Deploy a Sample Application to Monitor

Milestone 5: Access Grafana and Import Dashboards

Open http://localhost:3000 and log in with username admin and the password you set in Milestone 3.

Import two starter dashboards:

1. Click + → Import → enter dashboard ID 1860 (Node Exporter Full) → select Prometheus as the data source → Import.
2. Repeat with dashboard ID 15661 (Kubernetes Cluster Overview).

You now have real CPU, memory, disk, and cluster-wide dashboards without building anything from scratch.

Milestone 6: Write SLO Alert Rules with Burn Rate Logic

First, explore the raw metrics in Prometheus:

Open http://localhost:9090 and try these queries:

Remember

rate() calculates the per-second rate of change of a counter. [5m] means "look at the last 5 minutes." status=~"5.." is a regex matching any status code starting with 5 (500, 502, 503, 504).

Now define recording rules (pre-computed queries) and burn-rate alerts:

Common Mistake

Writing burn-rate alerts directly against raw metrics instead of recording rules forces Prometheus to scan millions of raw data points on every evaluation. Recording rules pre-compute the 5m/1h/6h/1d ratios once, on a schedule, so alert evaluation stays fast. This is not optional at any real scale.

Milestone 7: Configure Slack Alerts

1. Go to api.slack.com/apps → Create New App → From scratch.
2. Name it Monitoring-Alerts and select your workspace.
3. Click Incoming Webhooks → turn it on → Add New Webhook to Workspace.
4. Choose or create a channel (e.g. #alerts-dev) → Allow.
5. Copy the webhook URL — it looks like https://hooks.slack.com/services/T.../B.../xxx.

Common Mistake

Routing every severity into one Slack channel causes alert fatigue — engineers stop reading a channel that pages them at the same volume for both critical outages and minor warnings. Route severity: critical to a dedicated on-call channel, and severity: warning to a lower-priority channel reviewed during business hours.

Milestone 8: Build an SLO Dashboard in Grafana

In Grafana, go to Dashboards → New Dashboard, and add these four panels — each is one PromQL query pointed at your recording rules from Milestone 6.

Save the dashboard as Orders API — SLO Overview.

Tip

Export every hand-built dashboard as JSON (Dashboard Settings → JSON Model) and commit it to Git. Grafana stores dashboards inside its own pod storage — if that pod is deleted and recreated, unsaved dashboards are gone permanently.

Confirm the full stack is working end to end:

If everything above passes, your SLO alert rules will show as PENDING or FIRING (not INACTIVE) in the Prometheus Alerts UI once error rates actually cross the burn-rate thresholds.

Installing kube-prometheus-stack without setting resource limits will exhaust memory on a small local cluster. Prometheus holds metrics in memory before flushing to disk, and on a 2-core, 4GB Kind cluster the default configuration can consume all available memory within an hour. Always set explicit resource limits in your values file for local testing.

Writing burn-rate alert expressions directly against raw counters instead of recording rules forces Prometheus to scan enormous amounts of raw data on every evaluation cycle. Recording rules are required, not optional, once you move past a toy example.

Setting an SLO target of 99.999% without the engineering maturity to support it creates an error budget so small it's unusable — 99.999% only allows about 26 seconds of downtime per month, and any routine deployment can consume a meaningful chunk of that. Start at 99% or 99.5% and tighten the target only as reliability actually improves.

Routing all alerts, regardless of severity, into a single Slack channel trains engineers to ignore that channel entirely. Split critical (page-worthy) alerts from warning (next-business-day) alerts into separate channels from the start.

Not exporting Grafana dashboards as JSON means any manually built dashboard is lost the moment the Grafana pod restarts, since dashboards live in the pod's own storage by default.