It is 2 AM. Checkout is throwing 500s at Zerodha. Three engineers join the call. One starts restarting pods. Another starts querying the database directly on production. A third is messaging the CEO. Twenty minutes later nobody agrees on what actually broke, and one of the "fixes" just made it worse. This is not a technology failure. It is a process failure. The service was fine enough to run - the response was not. This module teaches the process senior SREs use so that an incident stays a single incident, not three. ### Why technical skill alone does not save you at 2 AM Under pressure, humans skip steps. You forget to check what changed. You start fixing before you understand blast radius. You make an assumption and never verify it. A structured process exists specifically to catch you doing this - it is scaffolding for a brain that is running on adrenaline and two hours of sleep. ### Where this fits in the reliability system you are building <cite index="2-70">Incidents are normal in complex systems.</cite> The measure of a good SRE team is not the absence of incidents - it is how fast they detect, how systematically they respond, and whether the same failure ever happens twice. This module connects directly to the SLOs module (an incident is, by definition, unacceptable burn against your error budget) and to the Kubernetes reliability module (many of the failure modes you will investigate live there). ---
Not every alert is a 2 AM phone call. A severity system exists so that everyone - engineering, support, leadership - agrees on how urgently to respond without arguing about it mid-incident. ### Why severity needs hard numbers, not vibes "This feels bad" is not a severity level. If severity is subjective, two people paged for the same incident will respond with different urgency, and your postmortem will spend more time on "should this have been a P1" than on the actual failure. **Severity** is a pre-agreed classification of incident impact, decided by concrete criteria - not a debate that happens during the incident. ### An example four-level severity model There is no single universal severity standard across SRE teams - some use SEV-1/2/3, some use P0-P4, some define the criteria differently. What matters is that your organisation picks one model and defines it with concrete criteria in advance. The table below is a common, usable example. | Level | Criteria | Response | |:---|:---|:---| | P0 | Total outage, data loss risk, or major revenue impact | Page immediately, IC required | | P1 | Significant feature broken, large user segment affected, SLO breach in progress | Page immediately, IC required | | P2 | Degraded experience, small user segment, workaround exists | Page during business hours or fast-follow | | P3 | Minor issue, no user impact yet, cosmetic or internal-only | Ticket, no page | > 📌 **Remember:** Organisations should define their own severity criteria > based on customer impact, business impact, data integrity, and urgency - > then apply those criteria consistently instead of debating severity mid-incident. ### A hypothetical example - a food-delivery platform Picture a food-delivery platform with a policy like the one above. Under that specific policy, a payment gateway affecting 40% of checkout attempts would likely be classified P0 - revenue impact, a large user segment, and a fast-burning error budget. A single restaurant's menu images failing to load would likely be P3. The actual classification always depends on total traffic, affected regions, duration, and whether a workaround exists - that is exactly why the criteria need to be written down in advance, not decided in the moment. ---
### What it is The **incident commander** (IC) is the one person who owns coordination during an incident - who is talking to whom, what has been tried, what happens next. The IC does not have to be the person who fixes the bug. ### Why this separation exists If the same person is debugging the database AND answering the CEO's Slack messages AND deciding when to declare "resolved," something gets dropped. Usually it is the debugging. Splitting "who coordinates" from "who fixes" lets the fixer stay heads-down while the IC manages everything else. ### When to use it A common and recommended policy is to name an IC within the first few minutes for any P0 or P1 incident. For P2/P3, one engineer can usually wear both hats. Adjust the threshold to what your team's incident volume and severity model actually need. > 🔴 **Common Mistake:** Assuming the most senior engineer on the call should > automatically be IC. The best IC is often not the best debugger - it is > whoever can stay calm, ask "what's the blast radius" repeatedly, and keep the > channel organised. Pick for the skill, not the seniority. ---
### What actually happens when an alert fires The instinct is to start fixing immediately. Resist it. The first five minutes should answer two questions before a single change is made: what is the blast radius, and what do the dashboards already say. ### Why touching things first makes debugging harder If you restart a service before checking dashboards, you have just erased the evidence you needed to diagnose the real problem. A restart can mask symptoms without fixing the cause, and now your metrics show a discontinuity that has nothing to do with the actual bug. ### The concrete first-five-minutes checklist * Define blast radius - what is broken, what is not, how many users affected right now * Open dashboards before touching anything - metrics first * Declare severity and name an IC if P0/P1 * Open a dedicated incident channel ```text ## Example first message in an incident channel (illustrative, fintech-style) [P1] Checkout error rate at 12%, started 02:14 IST IC: @priya Dashboards: <link> Fix channel: #inc-2847-fix ``` > **Note:** A dedicated fix channel separate from the main incident channel > keeps debugging chatter from drowning out status updates - covered next. ---
### What it is A predictable rhythm of updates - for example, internal updates every 15 minutes for a major incident, plus external status page updates in plain language - so that stakeholders are not repeatedly interrupting the IC to ask "any update?" ### Why cadence beats ad-hoc updates Without a cadence, silence gets interpreted as "nothing is happening" even when the team is heads-down on the actual fix. Establish a cadence appropriate to the severity and pace of the incident - a fast-moving P0 may need updates every 5 minutes, while a stable P1 investigation may only need one every 30. The point is predictability, not a specific number. ### War room discipline For larger incidents, separating coordination and status updates from detailed technical investigation can reduce noise and confusion. One common pattern: * Only the IC posts status in the main incident channel * Technical back-and-forth happens in a separate fix channel * Decisions are stated explicitly - "Decision: rolling back deploy #4471" - not implied Smaller teams sometimes use a single channel plus a pinned incident document instead - the specific mechanism matters less than keeping status updates separate from raw debugging chatter. > ⚠️ **Security:** Never paste customer PII, tokens, or credentials into an > incident channel, even temporarily for debugging. Redact before pasting logs. ### Internal vs external language Internal: "Redis connection pool exhausted after the 14:02 deploy, rolling back." External (status page, illustrative example for a live-streaming platform): "Some users may experience slow loading of live scores. We are actively working on a fix." ---
This is the core operational skill of this module. Every real incident, no matter the technology, follows the same six steps. +----------------------------------------------------+ | 1. Define blast radius | | 2. Check dashboards (metrics -> logs -> traces) | | 3. Correlate timeline against changes | | 4. Narrow scope (service/region/instance/customer) | | 5. Hypothesis -> test -> confirm or reject | | 6. Fix, verify, THEN declare resolved | +----------------------------------------------------+ ### Step 1 - Define blast radius **Blast radius** is the actual scope of what is broken - which services, which regions, which percentage of users. Get this before anything else, because it determines severity and who needs to be paged. ### Step 2 - Check dashboards, following the strongest signal A useful default order is metrics first, since they are fastest to read and show you *where* to look, then logs to see *what* is happening in that area, then traces to see *why* a specific request is slow. Treat this as a starting point, not a mandatory sequence - if a trace or a log line immediately points at the cause, follow that signal instead of working through the order mechanically. ### Step 3 - Correlate the timeline Line up the anomaly start time against recent deployments, configuration changes, dependency changes, and traffic shifts - these are the highest-value things to correlate against the incident timeline. Not every incident starts this way; some originate from certificate expiry, a cloud provider issue, slow resource exhaustion, or a latent bug with no recent trigger. Recent changes are simply the fastest thing to rule in or out first. > 💡 **Tip:** Keep a shared deploy log with exact timestamps. "What changed in > the last hour" should be a 10-second lookup, not a Slack archaeology dig. ### Step 4 - Narrow scope Is this every pod, one zone, one instance type, one customer, or truly systemic? A failure isolated to `ap-south-1b` points somewhere very different than a failure across every region. ### Step 5 - Hypothesis-driven investigation Form a specific, falsifiable hypothesis - "the new deploy added a synchronous call to a slow endpoint" - then test it before changing production. Guessing and changing things randomly is how one incident becomes three. ### Step 6 - Fix, verify, then declare resolved Make the smallest change that addresses the confirmed cause. Watch the metrics recover. Only declare resolved once the dashboard confirms it - not once the fix has been deployed. > 🔴 **Common Mistake:** Declaring an incident resolved the moment the deploy > finishes, without watching metrics actually recover. Metrics take time to > reflect reality - wait for confirmation, not hope. ---
It is 2 AM. Checkout is throwing 500s at Zerodha. Three engineers join the call. One starts restarting pods. Another sta...
Not every alert is a 2 AM phone call. A severity system exists so that everyone - engineering, support, leadership - agr...
What it is The incident commander (IC) is the one person who owns coordination during an incident - who is talking to wh...
What actually happens when an alert fires The instinct is to start fixing immediately. Resist it. The first five minutes...
What it is A predictable rhythm of updates - for example, internal updates every 15 minutes for a major incident, plus e...
This is the core operational skill of this module. Every real incident, no matter the technology, follows the same six s...
What a postmortem is A postmortem is the written record of an incident - what happened, why, and what will change so it ...
A team can run perfect incident response and still burn out if on-call itself is unhealthy. This section is about keepin...
This lab uses a single, consistent scenario throughout: a bad deployment introduces a memory leak, the pod gets OOMKille...
Concept Key fact Severity Define your own P0-P3 criteria in advance, apply them consistently IC role Coordinates, does n...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.