Imagine this scenario at a large trading platform. It is 3 AM. One payment-verification service slows down by 200ms. Nothing crashes. No alert fires for "service down." But within four minutes, eleven other services are timing out, the order-matching engine is rejecting trades, and the on-call engineer is staring at a dashboard that looks like everything broke at once. Nothing broke. Everything just got slow at the same time, and slowness spreads. This is the core difference between a single-machine bug and a distributed systems failure. A single machine either works or it does not. A distributed system has dozens of machines calling each other over a network that can delay, drop, duplicate, or reorder messages at any point. **Distributed systems** are a collection of independent machines that must coordinate to look like one reliable system to the user, even though the network between them is inherently unreliable. Most production outages at scale are not caused by bugs in business logic. They are caused by systems that were never designed to handle the ways a network can fail. > 📌 **Remember:** A distributed system is not "many computers." It is many computers pretending to be one, over a connection that can fail in more ways than the computers themselves can. ### Why single-machine intuition breaks you here On one machine, if a function returns, you know the result. In a distributed system, if a request times out, you genuinely do not know what happened. The request might have never arrived. It might have arrived and succeeded, with only the response lost on the way back. It might still be running right now. This uncertainty is not an edge case. It is the default condition every SRE designs around. ### The eight fallacies of distributed computing These are the famous eight fallacies of distributed computing - assumptions engineers make without noticing, and every one of them is false in production: * The network is reliable * Latency is zero * Bandwidth is infinite * The network is secure * Topology does not change * There is one administrator * Transport cost is zero * The network is homogeneous > 🔴 **Common Mistake:** Engineers write retry logic assuming "the network is reliable, this call will basically always work." Then a routine AWS `ap-south-1` AZ blip takes down a service that had zero resilience patterns because nobody planned for the network being the thing that fails. ---
### What CAP actually says When engineers hear "CAP theorem" they think it is academic. It is not. It is the single most practical lens for choosing a database or queue. **CAP theorem** states that during a network partition, a distributed data store can guarantee either Consistency (every read gets the latest write) or Availability (every request gets a response), but not both at the same time. Think of it like a bank with two ATMs in different cities that lose connection to each other for ten seconds. If both ATMs keep dispensing cash during that gap (available), someone could overdraw the account because neither ATM knows the other just paid out. If one ATM refuses to dispense cash until it reconnects (consistent), a customer stands there with a declined card even though they have money. * **P is not optional.** Partitions will happen - a network cable gets cut, an AZ loses connectivity, a router misbehaves. The real question during that partition is: do you refuse to answer rather than answer wrong (favor consistency), or do you keep answering with possibly stale data (favor availability)? PARTITION OCCURS | +------+------+ | | Consistency Availability (refuse to (answer with answer wrong) possible stale data) * This is a spectrum a system chooses *during a partition*, not a fixed label stamped on a product. The same database can behave differently depending on configuration, which operation you call, and which failure mode you hit. * **etcd** is the cleanest concrete example of choosing consistency during a partition - it uses Raft, and a minority partition simply stops accepting writes rather than risk two conflicting leaders. That is CAP's "C" side in action. * Many systems marketed as "AP" (like Cassandra or DynamoDB) can also be tuned toward stronger consistency for specific reads or writes, at a cost to availability or latency. Do not assume a product name tells you its CAP behavior - check the actual configuration and the specific operation. > 💡 **Tip:** Ask "if this data is 3 seconds stale, does anyone get hurt?" If yes (bank balance, payment status, inventory reservation), you need the stronger consistency guarantees. If no (a restaurant's average rating, a video view counter), you can favor availability and take the staleness. ### Consistency models beyond the CAP binary CAP is a simplification. Real systems pick a point on a spectrum: | Model | What it guarantees | Where it is used | |:---|:---|:---| | Strong consistency | Every read sees the latest write, everywhere | Payment ledgers, inventory locks | | Eventual consistency | Reads converge to latest write, eventually | Social feeds, product catalogs | | Read-your-writes | You always see your own writes immediately | User profile edits, comment posting | | Causal consistency | Related events stay in order across replicas | Chat apps, collaborative editing | > **Note:** "Eventually" in eventual consistency is not defined by the system - it depends on replication lag, which can be milliseconds or, during an incident, minutes. Never design a user-facing flow that assumes "eventually" means "instantly." ---
### Why distributed systems need consensus at all `etcd` backs every Kubernetes cluster's state. `Consul` and `ZooKeeper` back service discovery and locks across thousands of production systems. All three rely on the same underlying problem: how do multiple machines agree on one value, even if some of them crash or the network partitions? **Consensus** is the process by which a group of machines agree on a single value or ordering of events, despite some machines failing or messages being delayed. **Raft** is the algorithm most modern systems use to achieve this, chosen specifically because it is easier to reason about than its predecessor, Paxos. ### How Raft works, in plain terms Raft elects one node as **leader**. All writes go through the leader. The leader replicates every write to a majority of followers before confirming it back to the client. If the leader disappears, the remaining nodes hold an election and pick a new one. +-----------+ writes +-----------+ | Client |---------------->| Leader | +-----------+ +-----------+ | | | replicate to majority | | | +-----+ +-----+ +-----+ |Follower| |Follower| |Follower| +--------+ +--------+ +--------+ A write is only considered committed once a **majority** of nodes (not all) have it. This is why Raft and etcd clusters are always deployed with an odd number of nodes - 3 or 5. With 5 nodes, you can lose 2 nodes outright and the remaining 3 still form a majority to keep operating. But node loss is not the only way quorum disappears. A network partition can leave every node technically alive but unable to reach a majority - for example, a 5-node cluster split 2-and-3 by a network fault still has one side with a majority, but a 2-and-2-and-1 three-way split leaves no side with quorum at all, even though nothing crashed. > 📌 **Remember:** A 3-node etcd cluster tolerates the loss of 1 node, or any partition that still leaves 2 nodes able to talk to each other. A 5-node cluster tolerates 2. This is why `kubectl` starts failing mysteriously when a majority of etcd nodes can no longer reach each other - the cluster lost quorum and refuses to accept writes, by design, to avoid split-brain. ### What split-brain actually looks like **Split-brain** happens when a network partition causes two groups of nodes to each believe they are the legitimate leader, and both start accepting writes independently. When the partition heals, the two histories conflict and data is lost or corrupted reconciling them. Raft prevents split-brain by requiring a majority vote to become leader. A minority partition can never elect a leader, because it can never reach majority - it can only sit and refuse writes until it reconnects. This is Raft choosing consistency over availability during a partition, which is the CP side of CAP in action. ---
### How one slow service takes down eleven others A **cascading failure** is a chain reaction where the failure or slowness of one component causes the components depending on it to fail too, which causes their dependents to fail, and so on, until the outage is far larger than the original problem. This is exactly what happened in the 3 AM scenario at the start of this module. Here is the actual mechanism, step by step: 1. The payment-verification service slows from 50ms to 2000ms per call, but does not crash 2. Every service calling it now holds a thread or connection open for 2000ms instead of 50ms 3. Those callers hit their own connection pool limits, because pools are sized for fast calls 4. Requests to the callers start queueing, then timing out 5. Services calling *those* callers now see failures and start retrying 6. Retries add more load onto an already struggling payment-verification service 7. The retry storm makes payment-verification even slower, restarting the loop at a worse baseline The system did not have one point of failure. It had one point of *slowness*, and the absence of limits let that slowness propagate outward. > 🔴 **Common Mistake:** Teams design for "what happens if this service is down" and completely ignore "what happens if this service is just slow." Slow is worse than down - a down service fails fast and callers move on. A slow service ties up resources across the entire call graph. ### Thundering herd - the aftermath problem A **thundering herd** occurs when a large number of clients that were all waiting or retrying suddenly act at the exact same moment, overwhelming the system that is trying to recover. Picture Hotstar during an India-Pakistan match. The video CDN briefly errors out for 30 seconds. Every one of 2 million connected apps has a retry timer set to "retry in 5 seconds." All 2 million clients hit the CDN again within the same second, at the exact moment it was starting to recover, and knock it back down. The fix is **jittered exponential backoff** - instead of every client waiting exactly 5 seconds, each client waits a randomized amount that grows with each failed attempt. ```python import random import time def retry_with_jitter(attempt, base_delay=1, max_delay=30): """ Calculate a randomized backoff delay that grows with each attempt. Jitter prevents every client from retrying at the exact same instant. """ # Exponential growth: 1s, 2s, 4s, 8s... capped at max_delay exponential = min(max_delay, base_delay * (2 ** attempt)) # Full jitter: pick any value between 0 and the exponential delay # This spreads retries across a window instead of a single instant return random.uniform(0, exponential) for attempt in range(5): delay = retry_with_jitter(attempt) print(f"Attempt {attempt + 1}: waiting {delay:.2f}s before retry") time.sleep(delay) ``` > **Note:** "Full jitter" (randomizing between 0 and the max delay) generally spreads retry load more evenly than "equal jitter" (randomizing around the midpoint), which can reduce synchronized retry bursts. Neither strategy is a guarantee - the right choice depends on your specific traffic pattern and is worth testing under real load. ---
### Circuit breakers - stop calling what is already broken A **circuit breaker** is a pattern that stops a service from repeatedly calling a dependency that is already failing, giving the failing dependency room to recover instead of being buried under continued traffic. The analogy is literal - a home electrical circuit breaker trips and cuts power when it detects a dangerous overload, rather than letting the wiring keep burning. A software circuit breaker trips and stops outbound calls when it detects a dependency is failing, rather than letting failed calls pile up. A circuit breaker has three states: CLOSED --[failures exceed threshold]--> OPEN ^ | | [timeout elapses] | v +----[test call succeeds]------ HALF-OPEN | [test call fails] v OPEN * **Closed** - normal operation, all calls go through, failures are counted * **Open** - calls fail immediately without hitting the dependency at all, giving it breathing room * **Half-open** - after a cooldown, one test call is allowed through to check if the dependency has recovered > **Note:** The implementation below is a simplified educational version to show the state machine clearly. It is not production-ready - it tracks consecutive failures only (not failure *rate*), treats every exception the same as a slow call, and has no thread safety for concurrent requests. Production systems should use a mature library or a service mesh's built-in circuit breaking (like Istio's outlier detection) rather than hand-rolling this. ```python import time from enum import Enum class CircuitState(Enum): CLOSED = "closed" OPEN = "open" HALF_OPEN = "half_open" class CircuitBreaker: def __init__(self, failure_threshold=5, recovery_timeout=30): """ failure_threshold: consecutive failures before the circuit opens. recovery_timeout: seconds to wait before allowing a test call. """ self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.failure_count = 0 self.state = CircuitState.CLOSED self.opened_at = None def call(self, func, *args, **kwargs): # If open, check whether enough time has passed to try a test call if self.state == CircuitState.OPEN: if time.time() - self.opened_at > self.recovery_timeout: self.state = CircuitState.HALF_OPEN else: raise Exception("Circuit is OPEN - failing fast, not calling dependency") try: result = func(*args, **kwargs) except Exception as e: self.failure_count += 1 # Trip the breaker once failures cross the threshold if self.failure_count >= self.failure_threshold: self.state = CircuitState.OPEN self.opened_at = time.time() raise e else: # A successful call in HALF_OPEN closes the circuit again self.failure_count = 0 self.state = CircuitState.CLOSED return result ``` > ⚠️ **Security:** Never set `maxEjectionPercent` or the equivalent breaker threshold to 100% in a service mesh config. That allows every single instance of a dependency to be ejected at once, turning a partial outage into a complete one for anyone still trying to call it. ### Bulkheads - stop one dependency from starving everything A **bulkhead** isolates resources (connection pools, thread pools, memory) per dependency, so that if one dependency misbehaves, it can only exhaust its own allocated resources, not the resources shared by every other call the service makes. The name comes from ship design - a ship's hull is divided into watertight compartments so that a hole in one compartment floods only that section, not the entire ship. Without bulkheads, one shared connection pool serving both a critical payment API and a non-critical recommendation API means a slow recommendation API can consume every connection in the pool, starving payment calls that had nothing to do with the problem. ### Backpressure - telling upstream to slow down **Backpressure** is a signal a system sends to whatever is sending it work, indicating that it cannot keep up and the sender should slow down, buffer, or drop requests rather than keep pushing. Without backpressure, a struggling service just queues requests infinitely until it runs out of memory and crashes - the worst possible outcome, because it takes down a service that might have recovered if the load had simply eased off. Practical backpressure mechanisms an SRE configures directly: * HTTP 429 (Too Many Requests) with a `Retry-After` header * Bounded queues that reject new work once full, instead of growing forever * Concurrency limits and admission control at the API gateway * Rate limiting per client or per endpoint * Consumer flow control on message brokers (Kafka consumer lag-based throttling) > 🔴 **Common Mistake:** Using a Kubernetes `readinessProbe` as a general backpressure mechanism, by having it fail under high load to pull a pod out of rotation. Readiness is meant to signal "should this pod receive traffic at all," not "signal capacity pressure to shed load." Using it this way can cause load oscillation - pods go unready under load, remaining pods absorb more traffic, those become unready too, and total capacity collapses instead of gracefully shedding. Keep readiness probes focused on genuine health checks, and handle load shedding with the mechanisms above instead. ### Deadline propagation - so timeouts mean something **Deadline propagation** means passing a single, shrinking timeout value through every hop of a call chain, so a request that has already spent 4 of its 5 allowed seconds does not get another fresh 5-second budget at every downstream service. Without it, a client with a 5-second timeout can trigger a downstream chain of five services each independently allowing 5 seconds, so the client gives up while 20 seconds of wasted work continues downstream anyway. ```python import time def call_downstream(deadline_ts, service_call): """ deadline_ts is an absolute Unix timestamp, not a duration. Passing the same deadline forward (not a fresh timeout) means every hop respects the original caller's patience. """ remaining = deadline_ts - time.time() if remaining <= 0: raise TimeoutError("Deadline already passed - do not make this call") # Pass the same absolute deadline to the next hop, not a new fixed timeout return service_call(timeout=remaining, deadline=deadline_ts) ``` > 💡 **Tip:** gRPC has deadline propagation built in via context. If your stack is plain REST, you have to pass the remaining budget manually as a header (`X-Deadline` or similar) and have every service respect it. ### Idempotency - making retries safe An operation is **idempotent** if calling it multiple times produces the same result as calling it once. This matters because every resilience pattern above - retries, circuit breaker half-open tests, thundering herd recovery - assumes it is safe to call something more than once. Without idempotency, a retry after a timeout on a payment API can double-charge a customer, because the first call might have actually succeeded and only the response was lost. The standard fix is an **idempotency key** - the client generates a unique ID per logical operation and sends it with every attempt. The server stores which keys it has already processed and returns the original result instead of repeating the side effect. ```python def process_payment(idempotency_key, amount, account_id, seen_keys_store): """ seen_keys_store maps idempotency_key -> previous result. Retrying the same logical payment never charges twice. """ # If this exact operation was already processed, return the cached result if idempotency_key in seen_keys_store: return seen_keys_store[idempotency_key] # Only new keys reach the actual charge logic result = charge_account(account_id, amount) seen_keys_store[idempotency_key] = result return result ``` > 🔴 **Common Mistake:** Building retry logic on a payment or order-creation endpoint without an idempotency key. This pattern is a common cause of double-charge incidents in fintech systems generally, and it is entirely preventable with a UUID generated once per logical operation. ---
**Head-of-line blocking** happens when one slow request blocks other, unrelated requests behind it from being processed, because they are stuck waiting in the same queue or connection. HTTP/1.1 commonly processes requests sequentially per connection, so a slow response can block the requests queued behind it on that same connection, even though the server itself is free to handle them in parallel. (HTTP/1.1 technically supports pipelining, and browsers can open multiple parallel connections, but in practice a slow request commonly still blocks whatever else is queued on its own connection.) HTTP/2 fixed this at the protocol level with multiplexing - multiple requests share one connection without blocking each other. But head-of-line blocking can still happen one layer down, at the TCP level, if a single packet is lost and TCP holds up all the multiplexed streams waiting for retransmission. HTTP/3 (QUIC) solves this by moving multiplexing into a protocol built on UDP instead of TCP. > **Note:** If you see "request 1 is slow, and now everything queued behind it on the same connection is also slow" in an incident, check whether the client is HTTP/1.1 with a single connection before assuming the server is at fault. ---
Imagine this scenario at a large trading platform. It is 3 AM. One payment-verification service slows down by 200ms. Not...
What CAP actually says When engineers hear "CAP theorem" they think it is academic. It is not. It is the single most pra...
Why distributed systems need consensus at all etcd backs every Kubernetes cluster's state. Consul and ZooKeeper back ser...
How one slow service takes down eleven others A cascading failure is a chain reaction where the failure or slowness of o...
Circuit breakers - stop calling what is already broken A circuit breaker is a pattern that stops a service from repeated...
Head-of-line blocking happens when one slow request blocks other, unrelated requests behind it from being processed, bec...
Complete these steps in order. Each one builds on the last. Set up a Python virtual environment and install dependencies...
Pattern Problem it solves Where to apply it Circuit breaker Repeated calls to a failing dependency Any outbound call to ...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.