Learn the eight failure patterns that cause production incidents at scale — cascading failures, split-brain, thundering herds, and more — plus CAP theorem, circuit breakers, and the reasoning process that turns alert storms into structured diagnosis.
### A single server is simple to reason about When one server goes down, it is down. You restart it. Problem solved. A distributed system — ten services talking to each other, three databases, two caches, a message queue — fails in ways that are much harder to understand. Services can be partially up. Data can be inconsistent across nodes. One slow component can make everything else slow. A retry storm triggered by one failing service can take down three healthy ones. Most production incidents are not caused by a single component failing. They are caused by how multiple components interact when things go wrong. Understanding these interaction patterns is what separates an engineer who fixes symptoms from one who fixes root causes. ---
### 1. Cascading Failure One service fails. Everything that depends on it starts failing. Everything that depends on those services starts failing. The failure spreads like dominoes. ``` Payment API → depends on → Auth Service → depends on → Redis Cache ↓ Redis goes down ↓ Auth Service times out ↓ Payment API returns 503 ↓ Frontend shows error to all users ``` **Why it happens:** Services do not fail gracefully when their dependencies fail. They hold connections open, queue retries, and consume resources waiting for a response that never comes. **How to spot it in alerts:** Multiple services alerting at the same time, all starting within seconds of each other. The first alert is usually the root cause — everything after is downstream noise. **How to prevent it:** Circuit breakers (covered below) and timeouts on every external call. --- ### 2. Thundering Herd Many clients all try to do the same thing at exactly the same time, overwhelming a single resource. **Classic scenario:** Your cache goes down for 30 seconds. Every service that was reading from the cache now hits the database directly — simultaneously. The database, which was comfortably handling 100 queries/second from cache misses, suddenly gets 10,000 queries/second. It collapses. Now you have a database outage caused by a cache restart. ``` Cache restarts (30 second downtime) ↓ All 500 app servers get cache miss ↓ All 500 hit the database at once ↓ Database overwhelmed → goes down ↓ Cache is back but now database is down ``` **How to prevent it:** Cache stampede protection — add a small random jitter to cache expiry times so not everything expires at exactly the same moment. Use probabilistic early expiration. --- ### 3. Split-Brain Two nodes in a cluster both think they are the leader. Both accept writes. Now you have two versions of the truth and no way to know which is correct. **Classic scenario:** A network partition separates a database cluster into two halves. Each half loses connection to the other. Each half elects its own leader. Both accept writes for 10 minutes. Network reconnects. Now you have conflicting data. ``` Database Cluster: Node A, Node B, Node C Network partition happens: [Node A] ←— can't talk —→ [Node B, Node C] Node A thinks B and C are dead → elects itself leader → accepts writes Node B and C think A is dead → elect B as leader → accept writes 10 minutes of writes to both sides Network heals Now: conflicting data, undefined state ``` **How to prevent it:** Quorum-based consensus — a node can only become leader if it has agreement from a majority (quorum) of nodes. With 3 nodes, you need 2 to agree. A partition that isolates 1 node cannot elect a leader because 1 < quorum of 2. --- ### 4. Network Partition Services cannot reach each other even though both are running fine. From the outside it looks like a service is down — but both services are healthy, the network between them is not. **Why this matters for AIOps:** Your monitoring will alert on the symptom (service unreachable) not the cause (network partition). An AI agent that does not know about partitions will suggest restarting a service that is perfectly healthy. **How to detect it:** If service A cannot reach service B but service C can reach service B, the problem is the network path between A and B — not service B itself. --- ### 5. Partial Failure Some requests succeed, some fail, with no clear pattern. The service is up but degraded. This is harder to diagnose than a full outage. A full outage is obvious. Partial failure looks like flakiness — sometimes it works, sometimes it does not. **Common causes:** * One pod in a deployment is unhealthy — requests that land on it fail, others succeed * A database replica is lagging — reads from it return stale data * Memory leak on one instance — it handles requests slowly until it crashes and restarts **How to spot it:** Look at error rates broken down by pod or instance. If the error rate is 33% and you have 3 pods, one pod is likely the culprit. --- ### 6. Retry Storm A service fails. All callers retry. The retries overwhelm the service further. The service cannot recover because it is drowning in retries. ``` Service A gets slow (high latency) ↓ All callers timeout and retry ↓ Service A now handles 3x the traffic (original + 2 retries each) ↓ Service A gets slower ↓ More timeouts, more retries ↓ Service A goes down completely ``` **How to prevent it:** Exponential backoff — each retry waits longer than the last (1s, 2s, 4s, 8s). Add jitter — randomize the wait time slightly so not all callers retry at the same moment. --- ### 7. Resource Exhaustion A resource — database connections, file descriptors, memory, thread pool slots — runs out. Nothing can proceed until the resource is freed. **Classic scenario:** Your application creates a new database connection for every request but never closes connections properly. After 10,000 requests, you have 10,000 open connections. The database has a max of 100. New requests cannot get a connection. Everything hangs. **How to spot it:** Connection pool exhausted errors, thread pool full errors, out of memory errors. The metric to watch is not just the current value but the rate of change — if connections are growing and not shrinking, you have a leak. --- ### 8. Slow Dependency (Latency Injection) One upstream service becomes slow. Every service that calls it becomes slow. Every service that calls those services becomes slow. Latency propagates upstream through the call chain. This is subtler than a hard failure. The service is technically up and returning correct responses — just very slowly. Timeouts accumulate. Thread pools fill up waiting for responses. Eventually the service runs out of threads and starts rejecting requests. **How to prevent it:** Timeouts on every external call. Never wait forever. If a dependency takes more than N milliseconds, give up and return an error or a cached value. ---
### The pattern A circuit breaker sits between your service and a dependency. It monitors calls to that dependency. When too many calls fail, it "trips" — stops sending requests to the failing dependency and immediately returns an error instead. ``` CLOSED state (normal): Service → Circuit Breaker → Dependency (passes requests through) After 5 failures in 10 seconds → OPEN state: Service → Circuit Breaker → ✗ (returns error immediately, does not call dependency) (dependency gets time to recover) After 30 seconds → HALF-OPEN state: Service → Circuit Breaker → Dependency (sends one test request) if test succeeds → back to CLOSED if test fails → back to OPEN ``` **Why this stops cascading failures:** When a dependency goes down, a circuit breaker stops your service from holding connections open waiting for it. Your service fails fast instead of slow. Resources are freed. The dependency gets space to recover without being flooded with requests. ```python import time from enum import Enum class CircuitState(Enum): CLOSED = "CLOSED" # normal, requests pass through OPEN = "OPEN" # failing, requests blocked immediately HALF_OPEN = "HALF_OPEN" # testing, one request allowed through class CircuitBreaker: def __init__(self, failure_threshold=5, recovery_timeout=30): """ failure_threshold: how many failures before opening the circuit recovery_timeout: seconds to wait before trying again (HALF_OPEN) """ self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.failure_count = 0 self.state = CircuitState.CLOSED self.last_failure_time = None def call(self, func, *args, **kwargs): """ Wrap a function call with circuit breaker protection. Use this instead of calling the dependency directly. """ # OPEN: check if enough time has passed to try again if self.state == CircuitState.OPEN: time_since_failure = time.time() - self.last_failure_time if time_since_failure >= self.recovery_timeout: self.state = CircuitState.HALF_OPEN else: # Still open — fail fast, do not call the dependency raise Exception(f"Circuit is OPEN. Dependency unavailable. Retry in {self.recovery_timeout - time_since_failure:.0f}s") # CLOSED or HALF_OPEN: try the actual call try: result = func(*args, **kwargs) # Success — reset failure count, close the circuit if self.state == CircuitState.HALF_OPEN: print("Circuit recovered — back to CLOSED") self.failure_count = 0 self.state = CircuitState.CLOSED return result except Exception as e: # Failure — increment counter, check if we should open self.failure_count += 1 self.last_failure_time = time.time() if self.failure_count >= self.failure_threshold: self.state = CircuitState.OPEN print(f"Circuit OPENED after {self.failure_count} failures") raise e # Example: protecting a database call with a circuit breaker import requests db_circuit = CircuitBreaker(failure_threshold=3, recovery_timeout=10) def query_database(query): """Simulated database call.""" response = requests.get(f"http://database-service/query?q={query}", timeout=2) return response.json() def get_user_safe(user_id): """Database call protected by circuit breaker.""" try: return db_circuit.call(query_database, f"SELECT * FROM users WHERE id={user_id}") except Exception as e: # Circuit is open or call failed — return cached value or error return {"error": str(e), "cached": True} ``` ---
### What it says CAP theorem states that a distributed system can only guarantee two of these three properties at the same time: * **C — Consistency:** Every read returns the most recent write. All nodes see the same data at the same time. * **A — Availability:** Every request gets a response (not an error). The system is always up. * **P — Partition Tolerance:** The system keeps working even when network partitions occur (nodes cannot talk to each other). Network partitions happen in real distributed systems — cables get cut, switches fail, cloud availability zones lose connectivity. So in practice, **P is not optional**. Every distributed system that runs across multiple machines must tolerate partitions. This means the real trade-off is: **when a partition happens, do you choose Consistency or Availability?** ### CP systems: choose consistency over availability When a partition occurs, a CP system stops accepting writes (or all requests) rather than risk returning inconsistent data. **Example:** You have a 3-node database. A partition isolates one node. The isolated node refuses to serve requests because it cannot confirm it has the latest data. It is unavailable — but the data it does serve is always correct. **Real systems:** Zookeeper, etcd, HBase, MongoDB (in default config) **When to use CP:** Financial transactions, inventory counts, anything where two conflicting writes would cause a serious problem. ### AP systems: choose availability over consistency When a partition occurs, an AP system keeps accepting reads and writes on all nodes. When the partition heals, nodes sync and resolve conflicts. **Example:** Same 3-node setup, partition isolates one node. The isolated node keeps accepting writes. When the network heals, the system reconciles the conflicting writes using a merge strategy (last write wins, or application-defined merge logic). **Real systems:** Cassandra, CouchDB, DynamoDB (default), DNS **When to use AP:** Social media likes and view counts, shopping cart contents, anything where "eventually correct" is acceptable and staying available matters more than instant consistency. ### What this means for AIOps When you see these incidents, CAP explains what happened: | Incident | What Happened | |---------|--------------| | "Database is refusing writes" | CP system lost quorum during a partition — chose consistency over availability | | "Users see stale data for a few minutes" | AP system during or after a partition — chose availability, consistency will catch up | | "Split-brain: two leaders" | AP system without proper conflict resolution | | "Service degraded but not down" | System correctly shedding load to maintain partial availability | ---
### Timeout — never wait forever Every call to an external service must have a timeout. Without timeouts, one slow dependency can exhaust your entire thread pool. ```python import requests # BAD — waits forever if the service hangs response = requests.get("http://payment-service/process") # GOOD — gives up after 2 seconds response = requests.get("http://payment-service/process", timeout=2) ``` ### Retry with exponential backoff and jitter ```python import time import random def call_with_retry(func, max_retries=3, base_delay=1): """ Retry a function with exponential backoff and jitter. Exponential backoff: each retry waits twice as long as the last Jitter: adds randomness so not all callers retry at the same moment """ for attempt in range(max_retries): try: return func() except Exception as e: if attempt == max_retries - 1: raise # last attempt — give up and raise the error # Exponential backoff: 1s, 2s, 4s... delay = base_delay * (2 ** attempt) # Jitter: add random fraction of the delay # This spreads out retries across callers — prevents retry storm jitter = random.uniform(0, delay * 0.1) wait_time = delay + jitter print(f"Attempt {attempt + 1} failed. Retrying in {wait_time:.2f}s") time.sleep(wait_time) ``` ### Bulkhead — isolate failures In a ship, bulkheads are walls that divide the hull into sections. If one section floods, the others stay dry. In software: give different operations their own resource pools. If one pool is exhausted, others are not affected. ```python from concurrent.futures import ThreadPoolExecutor # BAD — all operations share one thread pool # A surge in payment processing exhausts threads for auth too shared_pool = ThreadPoolExecutor(max_workers=20) # GOOD — separate pools per service type # Payment surge cannot affect auth processing payment_pool = ThreadPoolExecutor(max_workers=10) auth_pool = ThreadPoolExecutor(max_workers=10) ``` ---
### A real incident pattern explained Incident: "The checkout service is down. 100% error rate. Started 14 minutes ago." Without these mental models: restart the checkout service. With these mental models — ask: ``` What changed 14 minutes ago? → Deployment? Config change? Traffic spike? Is it fully down or partially down? → Partial = one pod, one replica, one dependency → Full = all pods, likely a dependency failure What does the checkout service depend on? → Payment API, Auth Service, Database, Cache Are any of those alerting? → If yes → cascading failure, fix the root dependency first → If no → isolated checkout failure, look at the checkout service itself Is the error rate exactly 33% or 50%? → Suggests one of three or one of two pods is bad (partial failure) → Rolling restart may fix it Are there connection pool errors in the logs? → Resource exhaustion — look for a leak, not a restart Are there retry storms in the logs? → Add circuit breakers before restarting anything ``` This is the reasoning process. The mental models turn a chaotic alert storm into a structured diagnosis.
A single server is simple to reason about When one server goes down, it is down. You restart it. Problem solved. A distr...
1. Cascading Failure One service fails. Everything that depends on it starts failing. Everything that depends on those s...
The pattern A circuit breaker sits between your service and a dependency. It monitors calls to that dependency. When too...
What it says CAP theorem states that a distributed system can only guarantee two of these three properties at the same t...
Timeout — never wait forever Every call to an external service must have a timeout. Without timeouts, one slow dependenc...
A real incident pattern explained Incident: "The checkout service is down. 100% error rate. Started 14 minutes ago." Wit...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.