Learn to design resilient AWS architectures and control cloud spend - multi-AZ failure design, RTO/RPO, FinOps, and multi-account strategy.
Your CTO asks a simple question: "Can we survive an entire AWS Availability Zone going down during our biggest sale of the year?" You know the honest answer depends on decisions made months ago - decisions about which AZs your database replicas sit in, whether your Auto Scaling Group can actually absorb the load, and whether anyone has ever tested restoring from a backup rather than just taking one. This is what separates a cloud engineer from a cloud architect. A cloud engineer can deploy a VPC, an ALB, and an Auto Scaling Group correctly. A cloud architect can explain why the system survives when a component fails, what it costs to guarantee that, and where the organization is knowingly accepting risk to save money. Neither answer is right or wrong in isolation - the job is making the trade-off visible and deliberate instead of accidental. Every decision in this module - which AZs to use, which DR strategy to pick, which purchasing option to commit to, how many AWS accounts to run - is a trade-off between resilience, cost, and complexity. There is rarely one correct answer. There is only the answer that matches what your business actually needs. > 📌 **Remember:** Architecture and cost engineering are the same discipline viewed from two angles. Every resilience decision has a cost. Every cost-cutting decision has a resilience impact. Treat them together, not separately. ### Where This Module Fits in Your Cloud Engineering Path This module assumes you already know how to build the pieces - VPCs, EC2, Auto Scaling Groups, RDS, and IAM from AWS Core Services, plus the security controls from AWS Security Engineering. Here, you learn how to arrange those pieces so the whole system survives failure, recovers within a time your business can tolerate, and does not silently drain your AWS bill while doing it. ### The Well-Architected Framework as Six Lenses on One Decision Most engineers have heard of the Well-Architected Framework as a list of five or six pillars to memorize for a certification exam. That is the wrong way to use it. The Framework is a set of lenses you hold up against a real architecture decision to see what you might be missing. Operational Excellence -> Can you run and monitor this without heroics? Security -> Is access, data, and infrastructure protected? Reliability -> Does it recover from failure automatically? Performance Efficiency -> Are you using the right resources for the job? Cost Optimization -> Are you paying for exactly what you need? Sustainability -> Are you minimizing the environmental impact of your workload? Think of the six pillars like six different inspectors looking at the same building. The structural engineer checks if it will stand in an earthquake (Reliability). The accountant checks if it was built within budget (Cost Optimization). The security consultant checks who can get through the doors (Security). A "well-architected" building is not the one that maximizes every inspector's individual score - it is the one where every trade-off between them was made on purpose. ### A Real Trade-off Walked Through All Six Pillars Say you are deciding whether to run your database as a single RDS instance or Multi-AZ. * **Reliability** - Multi-AZ wins clearly. A single instance is a single point of failure. * **Cost Optimization** - Single instance wins. Multi-AZ roughly doubles your RDS cost. * **Operational Excellence** - Multi-AZ wins. Failover is automatic; a single instance means a 2 AM page and a manual restore. * **Performance Efficiency** - Roughly a tie for most workloads; the standby does not serve read traffic by default. * **Security** - Neutral; this decision does not change your attack surface. * **Sustainability** - Single instance wins narrowly; less duplicated compute running at all times. Notice this is not a math problem with one right answer. A startup's internal admin tool might reasonably choose the single instance to save money. A payments platform processing customer transactions cannot make that same choice - the cost of downtime dwarfs the cost of the standby. The Framework's job is making sure you chose deliberately, not by default. > 💡 **Tip:** In an architecture interview, when asked to justify a decision, naming the pillar you prioritized and the pillar you traded away shows more seniority than simply defending the decision as "correct." Say "we optimized for Reliability and accepted the Cost Optimization hit because downtime costs us more than the standby instance," not just "we used Multi-AZ because it's more available." ---
An Availability Zone is not a marketing term - it is a physically separate data center with its own power, cooling, and network connectivity, linked to other AZs in the same region by low-latency private fibre. When AWS says "run production across at least two AZs," they mean: assume one of these data centers can disappear, and your application should keep serving customers anyway. The mistake most engineers make is deploying "across two AZs" as a checkbox - one EC2 instance in each - without actually verifying what happens when one AZ genuinely goes dark. ### Walking Through an Actual AZ Failure Normal state: ALB routes traffic evenly across AZ-a and AZ-b ASG: 2 instances in AZ-a, 2 instances in AZ-b RDS: primary in AZ-a, standby in AZ-b AZ-a fails completely: -> The 2 EC2 instances in AZ-a stop responding to health checks -> ALB stops routing to them within its configured health check interval -> For an RDS Multi-AZ deployment, automatic failover can promote the standby in AZ-b to primary -> DNS for the RDS endpoint updates to point at the new primary -> ASG detects unhealthy instances, launches replacements in AZ-b or AZ-c What determines whether customers notice: -> Health check interval and threshold (how fast ALB detects the failure) -> Whether AZ-b alone has enough capacity to absorb 100% of traffic -> RDS failover time (commonly in the range of 60-120 seconds for a typical Multi-AZ deployment, but this varies by workload and is never guaranteed to be zero) -> Whether your application retries gracefully during the failover window The detection mechanism and the capacity headroom are what most teams forget to test. If AZ-b was only ever running at 50% of what AZ-a was handling, "surviving" an AZ failure on paper does not mean surviving it in practice - your remaining AZ falls over from the sudden doubled load. > 🔴 **Common Mistake:** Sizing each AZ's capacity for exactly half the normal load, assuming both AZs will always be healthy simultaneously. If AZ-a fails, AZ-b needs to absorb 100% of traffic, not 50%. Size your Auto Scaling Group's minimum and desired capacity per AZ to survive losing one AZ entirely, not just to handle normal load split two ways. ### Building the Recovery Procedure, Not Just the Architecture A resilient architecture without a documented recovery procedure is a resilient architecture nobody trusts during a real incident. For each critical component, you should be able to answer three questions in advance: 1. **What actually fails first** - is it the ALB losing targets, the database losing its primary, or the ASG failing to launch replacements fast enough? 2. **How do you detect it** - which CloudWatch alarm fires, and how long does detection take? 3. **What is the recovery procedure** - is it fully automatic, or does an engineer need to do something, and what exactly? A component where the answer to question 3 is "an engineer needs to manually promote the read replica" is not automatically bad - but it needs to be a known, accepted trade-off, documented in a runbook, not a surprise discovered during an actual outage. ---
Availability Zones protect you from a data center failure. **Disaster Recovery (DR)** strategy is about something bigger - what happens if an entire AWS *region* becomes unavailable. This is a different, more expensive, and more deliberate decision than Multi-AZ. ### The Four DR Strategies, Ordered by Cost and Recovery Speed Backup and Restore -> Cheapest. Backups exist in another region. Restore takes hours. -> RTO: hours to a day. RPO: depends on backup frequency. Pilot Light -> Core infrastructure (like a database) replicates continuously to the DR region, but is minimally sized and mostly idle. -> On disaster, scale up the DR region's compute and go live. -> RTO: tens of minutes. RPO: minutes. Warm Standby -> A scaled-down but fully functional copy of the whole stack runs continuously in the DR region, ready to take full traffic once scaled up. -> RTO: minutes. RPO: near-zero. Active-Active (Multi-Region) -> Both regions run at full production capacity simultaneously, serving live traffic right now. -> RTO: near-zero, often seconds. RPO: near-zero. -> By far the most expensive and complex to operate correctly. ### RTO and RPO Are the Two Numbers That Drive Every DR Decision **RTO (Recovery Time Objective)** answers: "how long can we be down before it seriously hurts the business?" **RPO (Recovery Point Objective)** answers: "how much data can we afford to lose, measured in time since the last good backup or replication point?" Example: An e-commerce checkout service RTO = 15 minutes -> we cannot be down longer than 15 minutes during a sale RPO = 1 minute -> we cannot lose more than 1 minute of order data Example: An internal analytics dashboard used twice a week RTO = 24 hours -> nobody urgently needs this restored same-day RPO = 24 hours -> losing a day of data is a minor inconvenience These two numbers, agreed with the business before an incident happens, are what determine which of the four DR strategies you can justify. A payments system with a 15-minute RTO cannot choose Backup and Restore - restoring from a snapshot alone takes longer than that. An internal dashboard with a 24-hour RTO would be wasting money on Active-Active. > 📌 **Remember:** RTO and RPO are business decisions, not engineering decisions. An architect's job is presenting the cost of each DR strategy against the business's actual tolerance for downtime and data loss - not silently picking the most technically impressive option. > 🔴 **Common Mistake:** Testing backups by confirming they completed successfully, but never actually running a full restore. A backup that has never been restored is a hypothesis, not a safety net. The only backup you can trust is one you have watched come back to life in a working environment. ### Automating Failover with Route 53 Health Checks Deciding on a DR strategy is only half the problem - something needs to actually detect the primary region is down and redirect traffic to the DR region automatically. **Route 53 failover routing** is the DNS-level mechanism that does this. Route 53 Health Check | Continuously probes an endpoint (e.g. https://api.example.com/health) | Healthy -> DNS resolves to Primary Region's endpoint Unhealthy -> DNS automatically resolves to Secondary Region's endpoint The health check itself deserves real design thought - a health check that only verifies "the web server responds" will report healthy even if the database behind it is completely unreachable. A good health check endpoint verifies the full critical path: can the application reach its database, its cache, and any downstream dependency it truly cannot function without. ```bash ## Create a health check against the primary region's endpoint ## --request-interval 30 = checked every 30 seconds ## --failure-threshold 3 = must fail 3 consecutive checks before marked unhealthy aws route53 create-health-check \ --caller-reference primary-region-health-2024 \ --health-check-config \ Type=HTTPS,ResourcePath=/health,FullyQualifiedDomainName=api-mumbai.example.com,RequestInterval=30,FailureThreshold=3 ``` > **Note:** `FailureThreshold=3` combined with a 30 second interval means Route 53 itself waits roughly 90 seconds of continuous failure before it marks the endpoint unhealthy and flips DNS to the secondary region. Setting this too low risks flapping between regions during brief network blips; setting it too high delays detection beyond what your RTO allows. > 🔴 **Common Mistake:** Assuming the health-check detection time is the same as what your customers actually experience. Detection time and client-experienced failover time are not the same thing - DNS resolver caching and your record's TTL mean some clients keep resolving to the old, unhealthy endpoint until their local cache expires, even after Route 53 has already switched. A low TTL on the record shortens this gap, but does not eliminate it entirely for every resolver. ---
### The Four Cost Categories Every AWS bill, no matter how complex it looks, breaks down into four categories. Recognizing which category a cost falls into is the first step to controlling it, because each category is optimized differently. Compute -> EC2, Lambda, Fargate, EMR - what you pay to run workloads Storage -> S3, EBS, EFS - what you pay to keep data sitting somewhere Data Transfer -> traffic between AZs, regions, or out to the internet Requests -> per-API-call charges - S3 requests, Lambda invocations, NAT Gateway processed bytes Most engineers focus their cost-cutting energy entirely on Compute, because it is the most visible line item, and miss that Data Transfer and Requests can silently dominate a bill at scale - especially in an architecture nobody designed with cost in mind from the start. ### Rightsizing with Compute Optimizer and CloudWatch Metrics The most reliable way to overspend on AWS is running instances sized for a peak load that happens twice a year, all year round. **Rightsizing** means matching instance size to actual, observed usage instead of a guess made at launch time. **AWS Compute Optimizer** analyzes your CloudWatch metrics history - CPU utilization, memory (if the CloudWatch agent is installed), network throughput - and recommends a better-fitting instance type, often showing the estimated monthly savings directly. EC2 instance running at 8% average CPU utilization for 30 days | Compute Optimizer analyzes utilization history | Recommendation: downsize from m5.2xlarge to m5.large | Estimated savings: ~$180/month on this instance alone ```bash ## Get rightsizing recommendations for all EC2 instances in the account aws compute-optimizer get-ec2-instance-recommendations ``` > 💡 **Tip:** Compute Optimizer needs at least 14 days of CloudWatch data before it can generate a confident recommendation. Do not rightsize an instance that just launched, and do not rightsize based on a single busy day - look at a full normal business cycle, including your peak periods, before downsizing anything. > 🔴 **Common Mistake:** Rightsizing purely on average CPU utilization while ignoring peak periods. An instance averaging 15% CPU but spiking to 95% every day at checkout time cannot be safely downsized just because the average looks low. Always check p95 or p99 utilization, not just the average, before committing to a smaller instance size. ---
AWS gives you several ways to pay for compute, and the right choice is entirely about how predictable your workload is - not about which option is "cheapest" in isolation. ### The Purchasing Options Compared | Purchasing option | Discount vs On-Demand | Commitment | Best for | |:---|:---|:---|:---| | On-Demand | None (baseline) | None | Unpredictable, short-lived, or brand new workloads | | Compute Savings Plans | Up to ~66% | 1 or 3 years, $/hour commitment | Steady baseline usage that may shift across instance types, regions, or even EC2/Fargate/Lambda over time | | EC2 Instance Savings Plans | Up to ~72% | 1 or 3 years, committed to a specific instance family in a specific region | Steady usage where the instance family is already known and unlikely to change | | Reserved Instances | Up to ~72% | 1 or 3 years, specific instance family | Very stable, long-running, specific instance types, often used for legacy commitments | | Spot Instances | Up to ~90% | None, generally interruptible with a two-minute notice when AWS is able to provide one | Fault-tolerant, interruptible batch or stateless work | **Savings Plans are not one single thing.** A **Compute Savings Plan** commits you to a dollar amount of compute spend per hour and automatically applies that discount across whatever instance family, size, region, OS, or even compute service (EC2, Fargate, Lambda) you actually use - maximum flexibility, slightly lower discount ceiling. An **EC2 Instance Savings Plan** commits you to a specific instance family within a specific region in exchange for a higher discount ceiling, but loses the flexibility to shift into a different family or service later. **Reserved Instances** are the oldest and least flexible of the three - tied to a specific instance type, and in some cases a specific Availability Zone, with the least room to adapt as usage changes. ### Building a Decision Framework You Can Defend Decision framework: Workload runs 24/7, steady, but instance types may shift over time -> Compute Savings Plan Workload runs 24/7, steady, on a known, unchanging instance family -> EC2 Instance Savings Plan Workload is stateless and can tolerate interruption -> Spot Instances Workload is short-term, new, or genuinely unpredictable -> On-Demand Legacy commitment already in place on a specific instance family -> Reserved Instances > 📌 **Remember:** Never commit to a 1 or 3 year Savings Plan or Reserved Instance before you have at least a few months of stable On-Demand usage data. Committing early, before you understand your actual steady-state usage, locks in a guess instead of a fact. ---
Some of the most expensive line items on an AWS bill are invisible until you know to look for them, because they come from architecture decisions made for entirely different reasons - security, simplicity, reliability - with a cost consequence nobody connected at the time. ### Cross-AZ Data Transfer Traffic between two resources in *different* Availability Zones within the same region is not free. A chatty microservice architecture, where dozens of services call each other constantly, can rack up meaningful cross-AZ transfer charges if those services happen to land in different AZs by chance rather than by design. Service A (AZ-a) <---chatty API calls---> Service B (AZ-b) -> Every call crosses an AZ boundary -> Billed per GB, both directions, silently accumulating ### NAT Gateway Processing Fees A NAT Gateway charges per hour it exists, *and* per GB of data it processes. Teams that route large volumes of data - like backup jobs or bulk S3 access from a private subnet - through a NAT Gateway can find this fee dwarfing the instance costs it supports. Private subnet instance downloading large datasets from S3 | Through NAT Gateway (charged per GB processed) | Out to S3 over the internet path > 💡 **Tip:** For supported services - specifically S3 and DynamoDB - a **VPC Gateway Endpoint** lets private-subnet traffic reach that service over the AWS network instead of through the NAT Gateway, avoiding its per-GB processing charge entirely. This does not extend to every AWS service; most others that need private connectivity use an Interface Endpoint (powered by PrivateLink) instead, which has its own hourly and per-GB cost. Adding a Gateway Endpoint for heavy S3 or DynamoDB traffic alone has cut some real production NAT bills by more than half. ### S3 Request Costs at Scale S3 storage itself is cheap, but requests are not free, and workloads that read or write millions of small objects - like a data lake with poor file batching - can find request costs exceeding storage costs entirely. > 🔴 **Common Mistake:** Assuming S3 is "basically free" because storage pricing looks negligible per GB, then discovering a Lambda function processing millions of tiny objects individually has generated a request-cost bill far larger than the storage bill for the same data. Batch small files before storing them where the access pattern allows it. ---
Your CTO asks a simple question: "Can we survive an entire AWS Availability Zone going down during our biggest sale of t...
An Availability Zone is not a marketing term - it is a physically separate data center with its own power, cooling, and ...
Availability Zones protect you from a data center failure. Disaster Recovery (DR) strategy is about something bigger - w...
The Four Cost Categories Every AWS bill, no matter how complex it looks, breaks down into four categories. Recognizing w...
AWS gives you several ways to pay for compute, and the right choice is entirely about how predictable your workload is -...
Some of the most expensive line items on an AWS bill are invisible until you know to look for them, because they come fr...
Setting Up Cost Anomaly Detection and Budgets Rightsizing and purchasing decisions control cost proactively. AWS Budgets...
A team that has never split into multiple AWS accounts often reaches instead for many VPCs inside one account to separat...
Cost and resilience meet directly in how you architect the flow of events through a system. Serverless, event-driven pat...
From Business Requirement to Final Architecture Every decision in this module - DR strategy, purchasing option, multi-ac...
The Three-Question Review Before Any Design Is Final Every real architecture review comes down to holding three question...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.