Learn the deployment and architecture patterns that make services reliable by design - graceful degradation, feature flags, blue-green and canary releases, multi-region strategy, RTO/RPO, and the Production Readiness Review checklist used before a service goes on-call.
Every module before this one taught you how to detect and respond to failure. This one teaches you how to **design systems so failure hurts less in the first place**. There is a limit to how much incident response and chaos testing can buy you. At some point, reliability has to be built into the architecture itself - how you deploy, how you fail over, how you decide a service is even ready to hold customer traffic. This module is about that layer of decisions, made before an incident ever happens. > 📌 **Remember:** Incident response makes failure survivable. Architecture makes failure less likely and less painful. You need both, but this module is about the second one. ### The Mental Model for This Module Reliability Architecture | +-- Reduce impact of failure | `-- Graceful degradation | +-- Reduce deployment risk | |-- Feature flags | |-- Blue-green | |-- Canary | `-- Dark launch | +-- Survive infrastructure/region failure | |-- Active-passive | |-- Active-active | |-- Replication (sync/async) | `-- RTO / RPO | `-- Prove production readiness `-- Production Readiness Review Every section below fits into one of these four buckets. Keep this map in mind - it's what turns five separate topics into one coherent way of thinking about reliability. ---
**Graceful degradation** means a service keeps working, in a reduced but still useful form, when one of its dependencies fails - instead of failing completely because one non-critical piece broke. Think about a food delivery app's home screen. It needs restaurant listings, prices, delivery estimates, and personalized recommendations. If the recommendation engine goes down, a badly designed app shows an error page for the entire home screen. A gracefully degraded app just shows a generic "popular near you" list instead of personalized picks - the user can still order food. Recommendation service healthy: Home screen -> personalized picks + restaurants + prices + delivery time Recommendation service down: Home screen -> generic popular list + restaurants + prices + delivery time (user can still complete the core task: order food) ### Designing for Degradation * **Identify the core user journey first.** For an e-commerce app, that's "browse, add to cart, pay." Everything else - reviews, recommendations, wishlists - is enhancement. * **Make every non-core dependency optional at the call site.** Wrap the call in a timeout and a fallback, never let a slow recommendation call block the page render. * **Decide the fallback in advance, not during the incident.** "Show cached data," "show a generic default," and "hide the section entirely" are all valid fallbacks - pick one per feature ahead of time. Graceful degradation isn't just "catch the error and show something else." Every non-critical dependency should have an explicit **timeout**, a **bounded retry policy** (not infinite retries - see the distributed systems module's coverage of retry storms), a **circuit breaker** for dependencies that fail often enough to warrant one, and a defined **fallback** (cached data, a generic default, or hiding the section). Skip any of these and the "graceful" part quietly disappears under load. > 🔴 **Common Mistake:** Treating every dependency as equally critical. If your payment service and your review-widget service both have the same timeout and retry policy, you have not actually decided what matters most to the user - you've just hoped nothing breaks. ### A Worked Example: Dependency Mapping Before you can design fallbacks, you need to see the whole dependency graph and classify each edge. For a checkout flow: Frontend | Checkout | +-- Payments +-- Inventory +-- Recommendations `-- Notifications | Dependency | Critical? | Timeout | Retry | Fallback | |---|---|---|---|---| | Payments | Yes | 2s | Limited, idempotent | None - fail the checkout, show a clear error | | Inventory | Yes | 1s | Limited | Serve last-known cached availability | | Recommendations | No | 300ms | None | Show a generic "popular items" list | | Notifications | No | Async | Queued retry | Fire-and-forget, retry later, never block checkout | This table is the actual design artifact - not the prose above it. Building one of these for every critical flow, before an incident forces you to discover it the hard way, is the real deliverable of "designing for degradation." ---
A **feature flag** (or feature toggle) is a runtime switch that turns a piece of functionality on or off without a new deployment. This solves a problem that predates most other patterns in this module: **deploying code and releasing a feature are two different events**, and conflating them is what makes rollbacks slow and risky. Without feature flags: Deploy = Release Bug found in new feature -> must roll back the entire deployment With feature flags: Deploy != Release New code ships dark (flag off) -> flip flag on for 1% -> bug found -> flip flag off instantly (no redeploy, no rollback, the bad code is still there but simply not executing) ### Progressive Exposure * **Percentage rollout** - enable for 1% of users, watch metrics, increase gradually to 100%. * **User segment targeting** - enable for internal employees first, then beta users, then everyone. * **Kill switch** - a flag whose only purpose is "instantly disable this feature if something goes wrong," checked before any risky code path runs. A feature flag can be one of the fastest ways to disable a risky application feature without redeploying, **provided the affected behavior is fully controlled by the flag**. It is not a universal rollback: the bad code may already have executed before the flag check, the flag service itself may be unavailable, database migrations tied to the release may already have run, or the new code may have corrupted state that a flag flip can't undo. A feature flag disables a code path going forward - it doesn't undo what already happened. > 🔴 **Common Mistake:** Letting feature flags accumulate forever. A flag that has been at 100% for six months with no cleanup plan is just dead conditional logic making the codebase harder to read - schedule flag removal as part of the rollout plan, not as an afterthought. ---
### Blue-Green Deployment Two identical production environments exist at once - **blue** (currently live) and **green** (the new version). Traffic switches atomically from blue to green once green is verified healthy. Before switch: Users -> Load Balancer -> [BLUE: v1.0] (live) [GREEN: v1.1] (idle, being tested) After switch: Users -> Load Balancer -> [GREEN: v1.1] (live) [BLUE: v1.0] (idle, kept for instant rollback) * Rollback is just switching the router back to blue - typically seconds, not minutes. * Downside: you need double the infrastructure capacity during the switch window, and database schema changes need to be backward-compatible with both versions simultaneously. ### Canary Releases Instead of an all-or-nothing switch, traffic shifts to the new version gradually, with monitoring used to determine whether the rollout should continue, pause, or roll back. This decision can be made manually by an engineer watching dashboards, or automated with a progressive delivery controller - mature setups automate it, but canary itself doesn't require automation to be valuable. Canary rollout: 5% traffic -> v1.1 -----> watch error rate & latency for 10 min | healthy? -+- yes -> increase to 25% | +- no -> automatic rollback to v1.0 * Catches problems that only appear under real production traffic patterns, at a fraction of the blast radius of a full rollout. * Needs good SLIs and automated analysis (see the capacity planning module's coverage of Flagger) to be trustworthy - a canary nobody is watching is just a slow, riskier version of a full rollout. ### Dark Launches New code runs against real production traffic, but its output is never shown to users - only logged and compared against the current system's output. This is how you test a rewritten pricing engine or a new search-ranking algorithm: run it silently alongside the real one, compare results offline, and only cut over once you trust the new path completely. > 💡 **Tip:** Blue-green answers "can I switch back instantly." Canary answers "how do I limit blast radius while rolling forward." Dark launch answers "how do I test new logic against real traffic without any user ever seeing it." They solve different problems and are often used together, not as alternatives to each other. ---
### Active-Active vs Active-Passive * **Active-passive** - one region serves all traffic; a second region sits ready, receiving replicated data but no live traffic, until a failover is triggered. Simpler to reason about, cheaper, but failover takes time and the passive region's readiness is only as good as your last failover drill. * **Active-active** - multiple regions serve live traffic simultaneously. Failure of one region can allow traffic to shift to healthy regions with very low interruption - but that outcome is not automatic. It depends on DNS/traffic-routing convergence time, connection draining, session and cache state, and how well data replication is designed for regional failure. Far more complex than active-passive: you now need a strategy for data consistency across regions that are both accepting writes. Active-passive: Region A (active) <--replication-- Region B (passive, standby) All traffic -> A A fails -> manual/automated failover -> traffic -> B (minutes of downtime) Active-active: Region A (active) <--sync/async replication--> Region B (active) Traffic split by geography -> both regions serving live requests A fails -> traffic shifts to B -> interruption depends on routing/state design > 🔴 **Common Mistake:** Building active-active without first deciding a data consistency strategy. Active-active without a real plan for concurrent writes across regions just means you've built a distributed system that can silently create conflicting data - which is often worse than the outage you were trying to avoid. ### Global Load Balancing Routes users to the nearest healthy region automatically, using DNS-based or anycast routing, combined with active health checks so a degraded region stops receiving new traffic without a human intervening. ### RTO and RPO - The Two Numbers That Drive Architecture * **RTO (Recovery Time Objective)** - how long can the system be down before it's unacceptable? "We can be down for 15 minutes." * **RPO (Recovery Point Objective)** - how much data can you afford to lose? "We can lose at most 30 seconds of writes." These two numbers, agreed with the business in advance, determine almost every architecture decision downstream. | Target | Possible approaches | Relative cost | |---|---|---| | RTO: hours, RPO: hours | Daily backups, manual restore | Low | | RTO: minutes, RPO: minutes | Automated failover, async replication | Medium | | RTO: seconds, RPO: near-zero | Active-active with synchronous or near-synchronous replication, among other approaches | High | > 📌 **Remember:** RTO and RPO are business decisions dressed up as engineering requirements. A payments ledger and an internal admin tool do not need the same RTO/RPO - and pretending they do wastes enormous engineering effort on the tool that didn't need it. ### Data Replication - Synchronous vs Asynchronous * **Synchronous replication** - the system waits for the required replica acknowledgement before considering a write committed, which reduces how much acknowledged data can be lost during failure - at the cost of added write latency, since you're waiting on a round trip to another region. * **Asynchronous replication** - a write is confirmed locally and replicated to other regions in the background. Fast writes, but a region failure can lose the last few seconds (or more) of unreplicated writes. Production databases such as Aurora Global Database, CockroachDB, and Spanner make different trade-offs among replication latency, consistency, availability, and recovery objectives - their underlying architectures differ significantly, so avoid treating them as interchangeable examples of "sync" vs "async." Choose based on the workload's actual RTO/RPO and consistency requirements, not on which product is trending. > 🔴 **Common Mistake:** Choosing "multi-region" as a checkbox goal without connecting it to an actual RTO/RPO requirement. Multi-region is expensive in engineering time and operational complexity - it should be a direct answer to "what does the business need," not a default architecture choice. ### Designing for Partial Availability When a non-critical service (say, search suggestions) is down but the core journey (browse and buy) still works, the system should be designed so the outage of the non-critical piece never takes down the whole application. This is graceful degradation applied at the architecture level, not just the code level - it means your service boundaries and dependency graph were designed with "what is core" in mind from the start. ---
A **Production Readiness Review (PRR)** is a systematic checklist a service must pass before it goes on-call and starts serving real production traffic. It exists so that "is this service actually ready" is answered with evidence, not a gut feeling on launch day. ### The Five PRR Domains **Observability** * SLOs defined for the service, with an agreed error budget policy. * Dashboards built covering the Four Golden Signals. * Alerts configured, using multi-window burn-rate alerting, and actually tested (fired in staging, confirmed someone gets paged). **Capacity** * Load tested against expected peak traffic, with headroom above that documented. * Autoscaling configured and verified to actually trigger under load. * Resource requests and limits set based on real usage data, not guesses. **Security** * Secrets managed through a secrets manager, never hardcoded or committed. * Network policies applied - the service can only talk to what it actually needs to. * RBAC configured correctly - no service account has more permission than its job requires. **Reliability** * Every external dependency explicitly declared, with a fallback or degradation plan for each. * Pod Disruption Budgets configured where appropriate to protect availability during voluntary disruptions - note a PDB alone can't make a single-replica workload highly available. * Timeouts configured on outbound calls, with additional resilience controls - bounded retries, circuit breakers, or bulkheads - applied where the dependency's failure pattern actually warrants them. **Operability** * Runbooks written for the top failure modes, and tested by someone who didn't write them. * On-call rotation assigned, with escalation policy configured. * Postmortem process agreed and understood by the team before it's needed. > 📌 **Remember:** A PRR is not a bureaucratic gate - it's the difference between finding out a service can't handle load during a controlled load test versus finding out during a Diwali sale traffic spike. PRR flow: New service built | v PRR checklist run across 5 domains | +----+----+ | | Gaps All green | | v v Fix gaps Service allowed | onto on-call rotation +----+ | v Re-review > 🔴 **Common Mistake:** Running a PRR once at launch and never again. Services drift - a dependency gets added six months later with no fallback, or a runbook goes stale as the architecture changes. Some teams re-run a lightweight PRR after major architecture changes, not just before the very first launch. ---
Every module before this one taught you how to detect and respond to failure. This one teaches you how to design systems...
Graceful degradation means a service keeps working, in a reduced but still useful form, when one of its dependencies fai...
A feature flag (or feature toggle) is a runtime switch that turns a piece of functionality on or off without a new deplo...
Blue-Green Deployment Two identical production environments exist at once - blue (currently live) and green (the new ver...
Active-Active vs Active-Passive Active-passive - one region serves all traffic; a second region sits ready, receiving re...
A Production Readiness Review (PRR) is a systematic checklist a service must pass before it goes on-call and starts serv...
Audit a sample service against the PRR checklist. Take a sample service spec (or a real service if you have one) and sco...
PRR checklist (one line per domain) Domain Pass criteria Observability SLOs + burn-rate alerts + tested dashboards Capac...
Skipping the PRR and letting a service go on-call without one is the most common and most expensive mistake in this modu...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.