You already know how to look at a dashboard. This module is about something harder: deciding **what number actually matters**, deciding **how wrong that number is allowed to be**, and building **alerts that page you at the right moment, not every moment.** Prometheus, Grafana, and Loki are tools. SLOs are a decision-making framework built on top of those tools. This module assumes you can already query a metric - it teaches you which metric to pick, how to turn it into a promise, and how to know when that promise is broken. > 📌 **Remember:** A dashboard tells you what happened. An SLO tells you whether it mattered. ---
An **SLI (Service Level Indicator)** is a single, precisely defined measurement of some aspect of the service your users actually experience. The most common SRE mistake is picking an SLI that is easy to measure instead of one that reflects user pain. ### The Three SLI Shapes * **Availability ratio** - good events divided by total valid events. Example: `successful_requests / total_requests` over 5 minutes. * **Latency percentile** - what fraction of requests complete under a threshold. Example: `p99 latency < 300ms`. * **Quality ratio** - what fraction of responses were correct/complete, not just "not an error". Example: a search API that returns HTTP 200 but zero results is not a quality failure by status code, but it is one by user experience. > 🔴 **Common Mistake:** Measuring "server did not crash" instead of "user got what they asked for". A checkout service returning HTTP 200 with an empty cart is not "available" from the user's point of view. ### Choosing SLIs for Different Service Types Request/response service (API) -> availability ratio + latency percentile Data pipeline / batch job -> freshness (data age) + correctness ratio Storage system -> durability + read/write latency Streaming service (Hotstar) -> startup time + rebuffering ratio, NOT uptime alone Think about Hotstar during an IPL final. "Is the server up" is meaningless if 40% of users see a spinning buffer wheel. The SLI has to be "percentage of playback sessions with zero rebuffering events in the first 10 seconds." ### The Right Measurement Point Where you measure changes what the SLI can see: * **Client-side (RUM)** - closest to real user experience, but noisy and harder to attribute to your service specifically. * **Load balancer / edge** - sees everything that reaches your infrastructure, including failures the application layer never gets to log. * **Application server** - clean and easy to instrument, but blind to anything that fails before the request reaches your code. * **Database** - useful for internal SLOs owned by a data team, but far removed from what the end user actually experiences. A backend service can report 100% success while the load balancer is silently dropping 5% of connections before they ever reach it. If you only instrument the application layer, you are blind to that 5%. > 💡 **Tip:** Measure as close to the user experience as practical, while keeping the measurement boundary aligned with the service's actual responsibility. Client-side (RUM) data can surface failures that never reach your gateway at all, but a backend team's SLO may intentionally measure only the part of the journey that team owns. There is no single "correct" layer - pick the one that matches what you are actually promising. ---
An **SLO (Service Level Objective)** is the target value or range for an SLI, over a defined time window. SLI: availability ratio (successful_requests / total_requests) SLO: 99.9% over a rolling 30-day window ### SLA vs SLO vs SLI - Do Not Confuse These | Term | What it is | Who sees it | Consequence of missing it | |---|---|---|---| | SLI | The raw measurement | Engineering | None directly - it's just data | | SLO | Internal target on the SLI | Engineering + product | Triggers error budget policy (freeze, postmortem) | | SLA | External contractual promise, usually looser than the SLO | Customers/legal | Triggers refunds, penalties, contract breach | > 📌 **Remember:** Organizations commonly set internal SLOs stricter than their external SLAs, so they have margin to detect and correct reliability problems before ever breaching a contractual commitment. If your SLA promises 99.9% to customers, an internal SLO around 99.95% is a common way to build that margin in. ### Rolling Window vs Calendar Window * **Rolling window** (e.g. "last 30 days, updated every minute") - smooths out the "reset to zero on the 1st" problem, always shows current reliability trend. Preferred by most modern SRE teams. * **Calendar window** (e.g. "this calendar month") - easier for stakeholder reporting and quarterly business reviews, but creates a weird incentive: a bad outage on the 30th barely dents the budget for that month, while an outage on the 2nd haunts you for 29 days. > 💡 **Tip:** Use rolling windows for alerting and engineering decisions. Use calendar windows only for the monthly report you send to leadership. ### How Strict Should the SLO Be? This is a business conversation, not a pure engineering one. Every extra "9" costs exponentially more engineering effort. | SLO | Allowed downtime per year | Allowed downtime per 30 days | |---|---|---| | 99% | 3.65 days | 7.2 hours | | 99.9% | 8.76 hours | 43.2 minutes | | 99.95% | 4.38 hours | 21.6 minutes | | 99.99% | 52.6 minutes | 4.32 minutes | | 99.999% | 5.26 minutes | 25.9 seconds | > 🔴 **Common Mistake:** Picking 99.999% because it "sounds premium." At that level you need multi-region active-active, zero-downtime everything, and a team on-call 24x7 purely for that one service. Most internal tools, admin panels, and even many customer-facing APIs are perfectly fine at 99.9%. Ask "what actually breaks for the user, and how often is that acceptable" before picking a number. ---
If your SLO is 99.9% over 30 days, your **error budget** is the remaining 0.1% - the amount of unreliability you are *allowed* to spend. Error budget = (1 - SLO) x total valid events in the window Example: SLO 99.9%, 10 million requests in 30 days. Error budget = 0.001 x 10,000,000 = 10,000 failed requests allowed For a request-based SLI this gives you a budget in "number of failed/invalid requests allowed." Not every SLO is naturally expressed as an event count (a latency SLO, for instance, is usually tracked as "% of requests exceeding threshold" rather than a spendable count) - the underlying idea is the same, but the units change with the SLI shape. This reframes reliability from an abstract virtue into a **spendable resource**. A risky deployment, a chaos experiment, a maintenance window - these all cost budget. If you have budget left, ship fast. If you don't, stop. > 📌 **Remember:** The error budget is a negotiation tool between SRE and product teams. It turns "should we deploy this risky change" into a math problem instead of an argument. ### Error Budget Policy - What Actually Happens at Each Threshold This is the part most teams skip, and it's the part that gives the SLO teeth. The table below is an **example policy**, not a universal SRE standard - actual thresholds and actions should be negotiated between engineering and product for each service. Some teams freeze earlier, some never freeze and instead reprioritize the backlog, some scale response by business hours or remaining calendar time in the window. | Budget consumed | Example action | |---|---| | 0-50% | Normal operations. Ship features freely. | | 50-75% | Increase caution. Flag risky deployments for extra review. | | 75-100% | Feature freeze on the affected service. Only reliability work and bug fixes ship. | | 100%+ (budget exhausted) | Mandatory postmortem. All non-critical deploys blocked until a remediation plan exists. | > 🔴 **Common Mistake:** Defining an SLO and never writing *any* error budget policy. Without a policy, the SLO is just a number on a dashboard nobody acts on. Having an agreed policy - whatever its exact thresholds - is what makes the SLO operational, not any specific number in the table above. Imagine a payment-gateway path at a fintech-style platform: an exhausted error budget there would typically mean no new payment methods ship and no UI experiments run until the reliability issue causing the burn is fixed. This is illustrative of the pattern, not a claim about how any specific company actually operates. ---
This is the single most important alerting pattern in modern SRE, popularized by Google's SRE Workbook. It solves one specific problem: **how do you alert on SLO risk without either paging too late or paging too often?** ### The Core Idea: Burn Rate **Burn rate** = how fast you are consuming your error budget, relative to the rate that would exactly exhaust it by the end of the window. Burn rate of 1x = you will exhaust the entire 30-day budget in exactly 30 days (expected pace) Burn rate of 10x = you will exhaust the entire 30-day budget in 3 days Burn rate of 100x = you will exhaust the entire 30-day budget in ~7.2 hours ### Why One Threshold Is Not Enough * A short, sharp spike (2 minutes of 100% errors) burns budget fast but resolves before a slow alert window even notices. * A long, low-grade problem (0.5% error rate for 3 days) never trips a short-window threshold, but quietly eats the whole month's budget. The fix: **use multiple time windows simultaneously**, each requiring a different burn rate to fire, and require the short window and a slightly longer window to *both* agree before paging - this cuts false positives from brief blips. ### A Common Multi-Window Pattern A widely used version of this pattern (popularized by Google's SRE Workbook) pairs a **long window** (how much budget has actually burned over a meaningful period) with a **short window** (confirming the burn is still happening right now, not something that already recovered). Both windows must exceed the burn-rate threshold together before the alert fires - that pairing is what makes the alert trustworthy enough to page a human. | Tier | Long window | Short window | Burn rate | Detects | Response | |---|---|---|---|---|---| | Fast | 1 hour | 5 minutes | 14.4x | Fast, severe outages | Page immediately | | Medium | 6 hours | 30 minutes | 6x | Medium-severity sustained issues | Page immediately | | Slow | 3 days | 6 hours | 1x | Slow, creeping degradation | Ticket, not page - review next business day | > 📌 **Remember:** As a common starting policy, fast and medium burn get paged immediately, and slow burn becomes a ticket reviewed calmly rather than a 3 AM page. The right severity for your team also depends on service criticality, business hours, and how much budget is already gone - treat this as a sensible default, not a law. ### PromQL for Multi-Window Burn Rate Alerts Assume SLO = 99.9% (error budget = 0.1%), and you track error ratio via a recording rule. ```yaml groups: - name: slo-burn-rate-fast rules: - alert: HighBurnRateFast expr: | ( sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) > (14.4 * 0.001) and ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) > (14.4 * 0.001) for: 2m labels: severity: page annotations: summary: "Fast burn rate - budget exhausted in ~2 days at this rate" - name: slo-burn-rate-medium rules: - alert: HighBurnRateMedium expr: | ( sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) ) > (6 * 0.001) and ( sum(rate(http_requests_total{status=~"5.."}[30m])) / sum(rate(http_requests_total[30m])) ) > (6 * 0.001) for: 15m labels: severity: page annotations: summary: "Medium burn rate - budget exhausted in ~5 days at this rate" - name: slo-burn-rate-slow rules: - alert: HighBurnRateSlow expr: | ( sum(rate(http_requests_total{status=~"5.."}[3d])) / sum(rate(http_requests_total[3d])) ) > (1 * 0.001) and ( sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) ) > (1 * 0.001) for: 1h labels: severity: ticket annotations: summary: "Slow burn - budget on track to exhaust before window ends" ``` The pattern with two time ranges ANDed together (a long window + a short window) is intentional and applies to all three tiers, including the slow one - it means "this isn't just a brief blip, it's been sustained." That combination is what makes these alerts trustworthy enough to page a human at 3 AM instead of firing on every transient spike. > 🔴 **Common Mistake:** Alerting only on the raw error rate without connecting it to burn rate math. "Error rate > 1%" sounds reasonable but is meaningless without knowing your SLO - for a 99% SLO service that's fine, for a 99.99% SLO service that's a five-alarm fire. ---
Google's SRE book defines four signals that provide a broadly useful starting point for understanding the health of almost any service. * **Latency** - time to service a request. Split successful vs failed latency separately - a fast error is not the same as a fast success. * **Traffic** - demand on the system (requests/sec, concurrent streams for Hotstar, transactions/sec for Zerodha). * **Errors** - rate of failed requests, explicit (HTTP 500) and implicit (HTTP 200 with wrong content). * **Saturation** - how "full" the service is - CPU, memory, connection pool, queue depth. The signal that tells you how close you are to falling over even before errors appear. Golden Signal Grid for any service: Latency | p50, p95, p99 by endpoint, split success/failure Traffic | req/sec, by region, by client type Errors | error rate, by error class (4xx vs 5xx vs timeout) Saturation | CPU %, memory %, connection pool %, queue depth > 💡 **Tip:** If you are instrumenting a brand-new service and don't know where to start, build these four panels first. They cover 80% of "is this service healthy" questions before you need anything fancier. ---
You already know how to look at a dashboard. This module is about something harder: deciding what number actually matter...
An SLI (Service Level Indicator) is a single, precisely defined measurement of some aspect of the service your users act...
An SLO (Service Level Objective) is the target value or range for an SLI, over a defined time window. SLI: availability ...
If your SLO is 99.9% over 30 days, your error budget is the remaining 0.1% - the amount of unreliability you are allowed...
This is the single most important alerting pattern in modern SRE, popularized by Google's SRE Workbook. It solves one sp...
Google's SRE book defines four signals that provide a broadly useful starting point for understanding the health of almo...
The observability module elsewhere covers Loki setup. Here we cover what SRE actually needs from logs: the ability to re...
Tracing answers a question metrics and logs can't: across 10+ microservices, which one is actually slow? The Trace / Spa...
Not every dashboard serves the same purpose. Building one giant dashboard with 40 panels for every audience is a common ...
This lab deliberately defines two separate SLOs for the same service, because a latency problem does not necessarily bur...
SLO target to downtime allowance (30-day window) SLO Downtime allowed / 30 days 99% 7.2 hours 99.9% 43.2 minutes 99.95% ...
Setting the SLO too high without a business reason is the most expensive mistake on this list - 99.999% forces multi-reg...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.