The Zerodha trading platform processed 4 million orders on a single IPL final day. Six weeks before that, a senior SRE sat in a design review for the new order routing service and asked one question: "What happens to in-flight orders if this service restarts mid-transaction?" The engineer who built it had not thought about it. The answer led to a fundamental change in how state was persisted. That one question, asked six weeks early, prevented a P0 that would have hit during the highest-traffic day of the year. This is what senior SREs do. They do not just respond to incidents - they prevent entire categories of incidents by being in the right room at the right time, asking the right questions, and having the technical vocabulary to influence design before the first line of code is written. The previous modules in this roadmap taught tools and practices for operating reliable systems. This module teaches the upstream skill: designing systems that are reliable from the start. Architecture review, failure mode analysis, Production Readiness Reviews, RTO and RPO design, multi-region architecture, and graceful degradation. ### Why design-time reliability is worth 10x more than operational reliability Every reliability problem caught at design time costs one hour of conversation. The same problem caught at code review costs one day of rework. Caught in staging, one week of firefighting. Caught in production, one month of incidents, postmortems, and remediation. A senior SRE who reviews designs and shapes architecture decisions pays for their salary in a single prevented P0. This is why the most effective SREs are not the fastest at incident response - they are the ones who make incidents rare.
A software engineer reads a design document looking for correctness. A senior SRE reads the same document looking for failure modes. These are fundamentally different questions. Software engineers are trained to think about the happy path. SREs are trained to think about the unhappy path first. ### The 5 questions every SRE asks in a design review Before reading any design document in detail, an SRE must be able to answer five questions. If the document does not address them, those are your comments. **Question 1: What is the SLO for this service?** If the design does not specify an SLO, it has no reliability requirement. Without a reliability requirement, there is no basis for any architecture decision - no way to know if two nines or four nines is appropriate, no way to evaluate whether adding a cache is worth the complexity. Ask this first. If the engineer does not have an answer, help them derive one before reviewing anything else. **Question 2: What is the critical path?** Every service has a critical path - the sequence of operations that must succeed for the core user action to complete. Everything else is optional. A Swiggy food order has a critical path: menu fetch, cart update, payment processing, order placement. Everything else - recommendations, promotional banners, loyalty points - is not on the critical path. Map it explicitly. Every component on it is a potential single point of failure. Every component off it is a candidate for graceful degradation. **Question 3: What are the single points of failure?** A **single point of failure (SPOF)** is any component whose failure causes the entire system to fail. Finding SPOFs is the most valuable thing you can do in a design review. Common SPOF patterns to look for: Single database instance -> Entire service fails if DB crashes Synchronous external call -> Service fails if external API is slow Shared cache with no fallback -> Service fails if cache is unavailable Single-region deployment -> Regional outage takes down everything Config loaded once at startup -> Cannot update config without restart **Question 4: What happens when each dependency fails?** Draw a dependency map. For each dependency ask: what does this service do if this dependency is unavailable? If the answer is "it fails immediately", that is a hard dependency. If the answer is "it degrades gracefully", that is a soft dependency. Hard dependencies on the critical path are existential risks. The design should have a plan for each one. **Question 5: How does this service get deployed and rolled back?** If the engineer has not thought about rollback at design time, they will not have a working rollback procedure at 3 AM during an incident. The rollback strategy is not a nice-to-have - it is a reliability requirement. ```bash ## Questions to put in design review comments for every external dependency ## 1. What is the timeout on this call? ## 2. What happens if it takes 30 seconds instead of 300ms? ## 3. What is the retry strategy? ## 4. Is there a circuit breaker? ## 5. What is the fallback if this is unavailable? ## For every database in the design: ## 1. Is this a single instance or replicated? ## 2. What is the failover time if the primary goes down? ## 3. What happens to in-flight transactions during failover? ## 4. What is the backup and recovery procedure? ## For every queue or async component: ## 1. What is the max queue depth before the producer blocks? ## 2. What happens to messages if the consumer is down for 2 hours? ## 3. Is message processing idempotent (safe to retry)? ``` ### A worked example - reading a design document as an SRE Here is a simplified design document. Read it as an SRE would: Design: PhonePe Notification Service The notification service receives events from the payment service via an HTTP call. It looks up the user's notification preferences from the preferences database, then sends the notification via the Twilio SMS API or Firebase push notification API. The service will be deployed as a single replica initially and scaled up based on load. An SRE reading this immediately sees five problems: * **Synchronous HTTP call from payment service** - If the notification service is slow, it slows down payment processing. Payments should never wait for notifications. This must be async via a queue. * **Single replica initially** - Any restart causes notification delivery failures. A node drain takes it to zero replicas. * **Two external API dependencies (Twilio and Firebase)** - Both are third-party services. What happens if Twilio is down? Does the notification fail silently? Get retried? * **Preferences database** - What happens if this is unavailable? Does the notification fail or fall back to a default? * **No mention of idempotency** - If the payment service retries the HTTP call, does the user get two SMS notifications? None of these are the engineer's fault - they were focused on the happy path. This is exactly why design reviews with SRE involvement exist.
**FMEA** is a systematic technique for identifying every way a system can fail before it does. It originated in aerospace engineering (NASA used it for Apollo) and is now standard practice in senior SRE work. For every component in your system, list every way it can fail, estimate the probability and impact, and identify what would detect it and how to mitigate it. ### Building an FMEA for a microservices architecture Create a worksheet with these columns. Probability and Impact are scored 1-5. Detectability is scored 1 (easy to detect) to 5 (hard to detect). | Component | Failure Mode | Prob | Impact | Detect | RPN | Mitigation | Status | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | Payment DB | Primary node crash | 2 | 5 | 2 | 20 | Multi-AZ, auto-failover | TODO | | Payment DB | Disk full | 3 | 5 | 4 | 60 | Storage autoscaling, alert at 80% | TODO | | Twilio API | HTTP 500 responses | 3 | 4 | 2 | 24 | Retry with backoff, fallback to Firebase | TODO | | Twilio API | Latency spike >5s | 3 | 4 | 3 | 36 | 500ms timeout + circuit breaker | TODO | | Redis cache | Connection pool full | 3 | 3 | 2 | 18 | Pool size increase, alert on saturation | DONE | | Redis cache | Cache eviction under load | 4 | 2 | 1 | 8 | Increase memory allocation | DONE | RPN = Probability x Impact x Detectability. Higher RPN = higher priority reliability investment. ```bash ## Use FMEA output to drive design conversations ## "Payment DB disk full" has RPN 60 - the highest risk item ## This means the design MUST address storage autoscaling before launch ## Check current disk usage trend for the database aws cloudwatch get-metric-statistics \ --namespace AWS/RDS \ --metric-name FreeStorageSpace \ --dimensions Name=DBInstanceIdentifier,Value=prod-payment-db \ --start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S) \ --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \ --period 3600 \ --statistics Minimum \ --region ap-south-1 ## Project when disk will run out based on weekly trend ## If FreeStorage went from 50GB to 40GB in 7 days, ## you have approximately 35 days of runway at current growth rate ## This is the concrete data for your design review comment ``` ### Using FMEA to make the business case for reliability investment After building the FMEA, sort by RPN descending. The top items are your highest priority reliability investments. This gives you a data-driven argument for why certain reliability work should happen before certain feature work. Present it as: "The Payment DB disk full scenario has RPN 60. If it happens, it is a total payment outage. We have no mitigation in place. This is our highest reliability risk and should be addressed before the feature launch." That conversation is much easier to have with a spreadsheet than with a gut feeling.
A **Production Readiness Review (PRR)** is a structured checklist-based review that every service must pass before it joins the on-call rotation. It is not a bureaucratic checkbox exercise - it is a reliability conversation that ensures the team has thought through every dimension of operating the service safely at 3 AM. The goal is not to block deployment. It is to ensure that when something goes wrong, the on-call engineer has the tools, knowledge, and runbooks to diagnose and fix it without causing more damage. ### The five PRR domains **Domain 1 - Observability:** The service must be observable before it is deployed. If you cannot see what it is doing, you cannot debug it when it breaks. * SLOs defined, approved by product stakeholders, and implemented as Prometheus recording rules * Four Golden Signals (Latency, Traffic, Errors, Saturation) visible on an on-call dashboard * Multi-window multi-burn-rate alerts configured and tested in staging * Structured logging with correlation IDs and trace IDs * Every alert has a runbook linked in the alert annotation **Domain 2 - Capacity:** The service must have been load tested before production. * Load tested at 2x expected peak traffic (peaks are always underestimated) * Resource requests and limits set based on VPA recommendations from the load test * HPA configured for stateless services, VPA for stateful * Database connection pool sized for peak load, measured during load test * Pod Disruption Budget set so at least one replica survives any voluntary disruption **Domain 3 - Reliability:** The service must handle failures gracefully. * Hard and soft dependencies documented * Circuit breakers configured for all hard external dependencies * Timeouts set on every external call - no unbounded waits anywhere * Retry logic with exponential backoff and jitter (not naive retry loops) * Graceful degradation implemented for every soft dependency * Rollback procedure documented and tested in staging **Domain 4 - Operability:** The on-call engineer must be able to operate the service without the author present. * Runbook written for every alert that can fire * On-call rotation assigned and trained on the service * Deployment procedure documented including expected duration and rollback trigger criteria * On-call handoff template populated with service-specific context **Domain 5 - Security:** Secrets in a secrets manager (not environment variables), network policies applied, RBAC minimal, container runs as non-root. ```bash ## PRR as a GitHub issue with checkboxes - create one per service pre-launch ## Example PRR tracking issue ## == OBSERVABILITY == ## - [ ] SLO defined: availability 99.9%, latency p99 < 500ms ## - [ ] Approved by: product-manager@company.com, eng-lead@company.com ## - [ ] Four Golden Signals dashboard: https://grafana.internal/d/xxx ## - [ ] Burn rate alerts: fast (1h/5%), medium (6h/2%) - tested in staging ## - [ ] Every alert has runbook URL in annotations ## == CAPACITY == ## - [ ] Load test report: https://internal/load-test/notification-svc-2026-08 ## - [ ] Tested at 2x peak: 20,000 RPS (expected peak: 10,000 RPS) ## - [ ] HPA: minReplicas=2, maxReplicas=20, targetCPU=70% ## - [ ] PDB: minAvailable=1 ## - [ ] DB connection pool: max=50 tested at peak RPS ## == RELIABILITY == ## - [ ] Dependencies mapped: payment-db (HARD), redis (HARD), recommendations (SOFT) ## - [ ] Circuit breaker: payment-db timeout 500ms ## - [ ] Graceful degradation: empty recommendations if svc unavailable ## - [ ] Rollback tested in staging: procedure takes 4 minutes, documented ## == OPERABILITY == ## - [ ] Runbooks: https://internal/runbooks/notification-service ## - [ ] On-call rotation: payments-oncall team added ## - [ ] Deployment: blue-green, 10 min rollout window documented ## == SECURITY == ## - [ ] Secrets in Vault (confirmed, not in ConfigMap or env vars) ## - [ ] NetworkPolicy applied: default deny, explicit allows documented ## - [ ] Running as UID 1000 (non-root, verified in pod spec) ``` ### Blocking vs non-blocking findings Not every PRR gap blocks deployment. Use a simple classification: * **Blocking** - cannot go live without this. Missing SLOs, missing runbooks, no timeout on external calls, secrets in environment variables. * **Non-blocking with deadline** - must be fixed within 30 days of launch. Missing full tracing coverage, single-AZ database, no PDB. * **Nice to have** - tracked but not required. Performance optimisations, additional observability coverage. This prevents PRR from becoming a process that delays useful features indefinitely. It also creates an explicit 30-day commitment for the team.
**RTO (Recovery Time Objective)** is the maximum acceptable time the system can be unavailable. **RPO (Recovery Point Objective)** is the maximum acceptable data loss in case of failure. These are business requirements, not technical metrics. ### Deriving RTO and RPO from business requirements The conversation with the business is not "what RTO do you want?" - they will always say zero. The conversation is: "what does one hour of downtime cost, and how much are you willing to spend to reduce that risk?" Example: Razorpay payment processing Business: What does 1 hour of payment downtime cost? Answer: ~1 crore INR in lost transaction fees + merchant trust damage Business: What would it cost to achieve 5-minute RTO? Engineering: Active-passive multi-region with warm standby = 1.5x infrastructure cost Business: What would it cost to achieve 30-minute RTO? Engineering: Single region with Multi-AZ failover = 1.2x infrastructure cost Decision: 5-minute RTO with active-passive multi-region is the right tradeoff given the cost of downtime vs the cost of the architecture This is a business conversation. The engineer provides options with costs and capabilities. The business makes the decision with full information. ### The RTO and RPO design matrix Different targets require fundamentally different architectures: | RTO | RPO | Architecture Required | Relative Cost | | :--- | :--- | :--- | :--- | | Hours | Hours | Single region, backups only | 1x | | 15-30 min | Minutes | Single region, Multi-AZ, read replicas | 1.3x | | 1-5 min | Seconds | Active-passive multi-region, async replication | 2x | | Under 1 min | Near-zero | Active-active multi-region, sync replication | 3-4x | ```bash ## Measure your ACTUAL current RTO - not what you assume it is START_TIME=$(date +%s) ## Simulate a database failover in staging (RDS example) aws rds failover-db-cluster \ --db-cluster-identifier staging-payments \ --region ap-south-1 echo "Failover initiated. Waiting for application to recover..." ## Poll health check until service recovers until curl -sf http://staging-payments.internal/health > /dev/null; do ELAPSED=$(( $(date +%s) - START_TIME )) echo " Not healthy yet... ${ELAPSED}s elapsed" sleep 5 done ACTUAL_RTO=$(( $(date +%s) - START_TIME )) echo "Actual RTO measured: ${ACTUAL_RTO} seconds" echo "Document this - it is probably different from what the team assumed" ## For PostgreSQL with streaming replication - measure current RPO psql -h prod-primary.internal -c \ "SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;" ## Example output: 00:00:02.341 ## This means your current RPO is approximately 2.3 seconds ## If the primary fails right now, you lose 2.3 seconds of committed transactions ``` > 🔴 **Common Mistake:** Teams regularly assume their failover takes "a few minutes" when the actual measurement is 18 minutes. This gap only surfaces during an incident. Run failover drills in staging, time them, and use those real measurements for architecture decisions.
Multi-region is where RTO and RPO requirements force the most expensive and complex decisions. Understanding the trade-offs is core senior SRE knowledge. ### Active-passive vs active-active **Active-passive**: One region handles all traffic. The second region runs in standby, receiving replicated data but not serving requests. On failure, traffic fails over to the passive region. Normal state: Users -> Route53 (100% traffic) -> Region ap-south-1 (primary) Data replicates to ap-southeast-1 (standby) Failure state: ap-south-1 fails Route53 health check detects failure (30s-60s) Traffic switches to ap-southeast-1 RTO = Route53 detection + DNS propagation (1-3 minutes typically) **Active-active**: Both regions serve traffic simultaneously. Each handles some users. Data is replicated bidirectionally. On failure, all traffic routes to the surviving region with no switchover needed. Normal state: Users in Mumbai -> Route53 latency routing -> ap-south-1 Users in Singapore -> Route53 latency routing -> ap-southeast-1 Data replicates bidirectionally Failure state: ap-south-1 fails Route53 health check detects failure All traffic routes to ap-southeast-1 automatically RTO = Route53 detection time (seconds, not minutes) ### The data consistency problem that most architecture documents skip Active-active introduces a fundamental dilemma that you must make explicit: **Synchronous replication**: Every write must be confirmed by both regions before the write is acknowledged to the user. RPO = zero (no data loss possible). But write latency = normal latency + round-trip time between regions (Mumbai to Singapore is approximately 40ms). For a payment service processing thousands of writes per second, this added latency is often unacceptable. **Asynchronous replication**: Writes are acknowledged immediately after the primary persists them. The secondary region receives the data in the background. Write latency is unaffected. But if the primary fails before replication completes, those writes are lost. RPO = current replication lag (typically 1-30 seconds). Synchronous - zero data loss, higher write latency: User writes data -> Primary region persists -> Data sent to secondary -> Secondary confirms receipt -> Write acknowledged to user Extra latency: 40ms per write (round trip to secondary) Asynchronous - potential data loss, normal latency: User writes data -> Primary region persists -> Write acknowledged to user immediately -> Data sent to secondary in background Extra latency: zero Risk: if primary fails before background sync, those writes are lost For most services, asynchronous replication with a well-understood and business-approved RPO is the right choice. For payment systems, the decision requires explicit business sign-off on the RPO. ```bash ## Aurora Global Database - AWS managed multi-region ## Typically achieves 1 second replication lag (RPO = ~1 second) ## Create Aurora Global Cluster in primary region aws rds create-global-cluster \ --global-cluster-identifier payments-global \ --engine aurora-postgresql \ --engine-version 14.9 \ --region ap-south-1 ## Add secondary region cluster aws rds create-db-cluster \ --db-cluster-identifier payments-ap-southeast-1 \ --engine aurora-postgresql \ --global-cluster-identifier payments-global \ --region ap-southeast-1 ## Monitor replication lag - this is your live RPO measurement aws cloudwatch get-metric-statistics \ --namespace AWS/RDS \ --metric-name AuroraGlobalDBReplicationLag \ --dimensions Name=DBClusterIdentifier,Value=payments-global \ --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \ --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \ --period 60 \ --statistics Average \ --region ap-south-1 ```
The Zerodha trading platform processed 4 million orders on a single IPL final day. Six weeks before that, a senior SRE s...
A software engineer reads a design document looking for correctness. A senior SRE reads the same document looking for fa...
FMEA is a systematic technique for identifying every way a system can fail before it does. It originated in aerospace en...
A Production Readiness Review (PRR) is a structured checklist-based review that every service must pass before it joins ...
RTO (Recovery Time Objective) is the maximum acceptable time the system can be unavailable. RPO (Recovery Point Objectiv...
Multi-region is where RTO and RPO requirements force the most expensive and complex decisions. Understanding the trade-o...
Graceful degradation means the system continues providing the most important parts of its service even when some compone...
How a service is deployed is as important as how it is designed. The best-designed service can be made unreliable by a d...
This lab conducts a complete reliability architecture review, builds an FMEA, and implements PRR fixes on a real Kuberne...
Architecture review checklist Area What to check Red flag SLO Defined and stakeholder-approved? No SLO defined at all Cr...
Skipping the SLO conversation at design time and adding monitoring after launch is the most common reliability mistake s...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.