Kubernetes SLO Alerting with Prometheus Burn Rates

This milestone gives you a cluster with kube-prometheus-stack running. If you finished the monitoring project, reuse that cluster and skip to Milestone 2. Otherwise, run the condensed setup below.

Create prometheus-values.yaml. Retention is 30 days because your SLO window is 28 days, and Prometheus cannot compute a 28-day ratio from 15 days of data.

Common Mistake

Keeping the default 15-day retention with a 28-day SLO. The long-window queries silently return a ratio computed from only the data that exists, so your budget numbers look better than reality.

This milestone deploys the service you will measure. An SLO needs a request counter with a status code, so the app must expose one. A plain nginx container does not, so this uses a small example app that publishes http_requests_total{code, method}.

Its / path returns 200 and its /err path returns an error, which gives you a controllable source of failures. A ServiceMonitor tells Prometheus to scrape it.

Check which status code /err returns, because the SLI in the next milestone depends on it:

In recent versions /err returns 404 rather than 500. That is why this project defines a failed request as any response outside the 2xx range, instead of only 5xx. Confirm the target shows as UP at http://localhost:9090/targets (port-forward svc/kube-prometheus-stack-prometheus 9090:9090 in the monitoring namespace).

Common Mistake

Adding prometheus.io/scrape annotations and expecting scraping. Those annotations only work with a Prometheus config that reads them. With the Prometheus Operator, you create a ServiceMonitor.

This milestone turns a vague goal into a measurable promise. A Service Level Indicator (SLI) is the measurement, and a Service Level Objective (SLO) is the target for it. Writing them down first stops you from alerting on whatever metric happens to be available.

Test the SLI in the Prometheus UI before you build rules on it:

The sum matters. Without it you get one ratio per pod, and a pod with no errors hides a pod with many. The query returns nothing when there is no traffic, which is correct: a service nobody calls has no measurable error ratio.

Common Mistake

Setting an SLO of 99.999 percent. That allows about 26 seconds of failure per month, and one routine deployment can use most of it. A budget that small gets ignored.

This milestone pre-computes the error ratio over each time window. Burn-rate alerts compare a long window against a short one, so you need the same ratio at several lengths. Recording rules calculate each one on a schedule, so alert evaluation reads a stored value instead of scanning raw samples every time.

Create slo-recording-rules.yaml:

Tip

A real team would generate these rules from an SLO definition with a tool such as Sloth or Pyrra instead of writing eight copies by hand. Write them by hand once so you understand what those tools produce.

This milestone creates the alerts. Each one fires only when a long window and a short window both exceed the same burn rate. The long window proves the problem is significant, and the short window proves it is still happening, so the alert also stops quickly once the problem is fixed.

The threshold for a burn rate is the burn rate multiplied by the budget (0.01):

Create slo-alert-rules.yaml:

Cause-based alerts such as high memory or crash looping are not in this set on purpose. Page a person for symptoms users feel. Keep cause-based signals on dashboards, or as tickets, to help diagnose.

Common Mistake

Using one window per alert. A single short window pages for every blip. A single long window keeps paging for an hour after you fixed the problem. The pair is what makes the alert both sensitive and quick to clear.

This milestone sends critical alerts to one channel and warnings to another. If both land in one channel, people learn to ignore it, because a 2am outage and a slow leak look the same.

Create two Slack incoming webhooks (or one webhook per channel) and store them as a Secret so they never reach Git:

Create alertmanager-values.yaml. It uses api_url_file, which reads the webhook from the mounted Secret:

The default message already shows the alert name, labels, and annotations, so this config avoids custom Go templates. Custom templates inside Helm values need extra escaping, because Helm tries to render them first.

Security

Treat webhook URLs like passwords. Anyone holding one can post to your channel.

This milestone makes the error budget visible to people who do not read PromQL. In Grafana, create a new dashboard with these panels, each backed by your recording rules:

Save it as Orders API SLO Overview, then export the JSON from Dashboard Settings and commit it to Git. Grafana keeps dashboards in its own storage, so a recreated pod loses any dashboard that only lives in the UI.

This milestone records what the team does as the budget runs down. An SLO that changes no decisions is decoration. Create a one-page SLO.md in your repository with the SLO from Milestone 3 and a policy such as this:

Adjust the numbers for your own service, and state who decides when the policy is invoked. This document is the deliverable that interviewers ask about most.

First confirm the pieces exist:

Then break the service on purpose. This traffic generator sends one good and one bad request per loop, so about half the requests fail:

• In Prometheus, orders_api:error_ratio:rate5m should settle near 0.5.
• In a fresh lab there is little history, so the 1h ratio also reads high quickly. Open Alerts in Prometheus and watch OrdersAPIBurnRateCritical1h move from Inactive to Pending, then Firing after its two-minute for period.
• Confirm the message reaches your page channel, and that warning alerts go to the ticket channel.
• On the dashboard, confirm the burn rate panel shows a value far above 14.4 and the budget remaining panel drops.

Now fix the problem and watch the alerts resolve:

Record the time from the first bad request to the page, and the time from the fix to the resolved message. If the gap is longer than you want, shorten the for durations and note the trade-off.

Clean up when you finish:

Common Mistake

Testing with a service that has no traffic. A ratio of zero requests divided by zero returns nothing, so the alert never fires. Pair your SLO alerts with a separate check for a service that has stopped receiving traffic at all.

As an extension, add a latency SLO using the app's request duration histogram, for example "95% of requests complete in under 300 ms", and give it its own burn-rate alerts.