### The problem with guessing during incidents It is 3 AM. An alert fires. Error rate on the payment service jumped from 0.1% to 28% six minutes ago. Your on-call engineer opens the monitoring dashboard, sees a wall of red, and starts guessing. They restart a pod. Nothing changes. They check the database. Looks fine. They restart another pod. Still broken. Twenty minutes later the real cause turns out to be a Redis connection pool setting that was changed in a deployment two hours ago. This is how most incidents go without a structured approach. Engineers guess, try random fixes, and get lucky eventually. The guessing costs time. Time costs money. During peak hours on a platform like Razorpay or Zerodha, every minute of downtime has a direct financial cost. **Root Cause Analysis (RCA)** is the practice of finding the actual underlying cause of a failure — not just stopping the immediate symptoms. RCA during a live incident means following a structured path through your observability data instead of guessing. ### Why most debugging fails Most engineers debug by starting with whatever is most visible. The dashboard shows high error rates, so they look at the service throwing errors. That service looks unhealthy, so they restart it. The restart fixes the errors temporarily, but they come back. They never found the root cause — they just masked a symptom. Three things make production debugging hard: First, the symptom and the cause are almost never in the same place. A payment service throwing 503 errors might be caused by a Redis cache that is running out of memory, which was caused by a query pattern change in a deployment that happened two hours ago. The error is in the payment service. The cause is in Redis. The trigger is in a code change. Second, modern systems generate enormous amounts of data. Logs, metrics, traces, events — all happening simultaneously across dozens of services. Without a systematic approach to filtering this data, you are looking for a needle in a haystack while the haystack is on fire. Third, pressure during incidents makes clear thinking harder. A structured process removes the need to think about what to do next. You follow the steps. ### What this module teaches By the end of this module you will have: * A five-step RCA process that works for any production incident * A mental model for reading logs, metrics, and traces together * Real commands for Kubernetes and Prometheus-based debugging * A Python RCA assistant that pulls data from all three sources automatically * A postmortem template that captures root cause, not just what happened ---
### Why a fixed process matters under pressure When you are under pressure at 3 AM, cognitive load is high. A fixed process means you do not spend mental energy deciding what to do next — you follow the steps. The process also ensures you do not skip important checks in a rush to apply a fix. The five steps work together in sequence. Each step narrows the search space before the next one begins. The five steps are: STEP 1: Establish the timeline What changed just before the incident started? | v STEP 2: Identify the blast radius What is affected? One service, many services, one region? | v STEP 3: Read the signals Logs + metrics + traces together — not separately | v STEP 4: Form and test hypotheses Generate 2-3 possible causes, test each with one command | v STEP 5: Confirm root cause and fix Verify the fix resolves the signal before closing Never skip to step 5 before completing steps 1-4. The most common mistake in incident response is jumping to a fix before confirming the root cause. ### Step 1 — Establish the timeline Before touching anything, answer: **what changed just before the incident started?** Systems do not spontaneously develop new failure modes. Almost every incident is caused by a change — a deployment, a configuration update, a traffic spike, a scheduled job, a certificate expiry, a dependency update. ```bash ## Check recent deployments across all namespaces kubectl rollout history deployment --all-namespaces ## Check what changed in the last 2 hours in the payment namespace kubectl get events -n payment --sort-by='.lastTimestamp' | tail -20 ## Check recent pod restarts — a pod restarting repeatedly is a signal kubectl get pods -n payment --sort-by='.status.containerStatuses[0].restartCount' ## Check if any ConfigMaps changed recently kubectl get configmap -n payment -o yaml | grep -A2 "creationTimestamp" ``` Build a timeline in your head or on a notepad: ```text 14:28 — Deployment of payment-api v2.4.1 pushed 14:30 — No alerts 14:32 — Error rate starts climbing (first alert at 14:32) 14:35 — Error rate at 28%, PagerDuty fires 14:37 — You are paged ``` The 4-minute gap between deployment and first alert is your first clue. Something in the deployment caused a slow degradation, not an instant crash. > 📌 **Remember:** The gap between cause and symptom is almost always between 0 and 30 minutes. If an incident starts at 14:32, look for changes between 13:00 and 14:32. Changes older than 2 hours are rarely the cause. ### Step 2 — Identify the blast radius **Blast radius** means: how much is affected? One pod? One service? All services in one region? Everything? This tells you whether you are dealing with an isolated component failure or a systemic issue. ```bash ## Check error rates across all services — which ones are elevated? ## This query shows services with >1% error rate in the last 5 minutes ## rate() calculates per-second rate over a time window ## sum by (service) aggregates across all instances of each service ## In Prometheus query (run in Grafana or prometheus UI): ## sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) ## / ## sum(rate(http_requests_total[5m])) by (service) ## > 0.01 ## Check which pods are unhealthy right now kubectl get pods --all-namespaces | grep -v Running | grep -v Completed ## Check if multiple services started erroring at the same time kubectl get events --all-namespaces --sort-by='.lastTimestamp' | grep -i error | tail -30 ``` What the blast radius tells you: | Blast Radius | Most Likely Cause | |-------------|------------------| | One pod | Pod-level issue — memory leak, crash loop, bad config mount | | One service, all pods | Service-level issue — bad deployment, broken dependency | | Multiple services, same cluster | Shared dependency — database, cache, message queue | | Multiple services, multiple clusters | Infrastructure issue — network, DNS, cloud provider | | One region only | Regional issue — cloud zone, CDN, regional config | If the blast radius is multiple services, you are almost certainly looking at a shared dependency failure. Stop looking at individual services and go find the dependency. ### Step 3 — Read the signals together This is the most important step. Most engineers read logs OR metrics OR traces. The answer is almost always in the intersection of all three. Think of it like a doctor diagnosing a patient. Symptoms (metrics) tell you something is wrong. Patient history (logs) tell you what happened. Test results (traces) tell you exactly where the problem is. Metrics Logs Traces (what is wrong) (what happened) (where it broke) | | | v v v Error rate at 28% "connection pool Request to /payment Latency at 8200ms exhausted" repeated took 8s, stuck at DB connections=50 every 200ms redis-cache step | | | +--------------------+----------------------+ | v Root cause: Redis connection pool exhausted after deployment changed pool size from 100 to 10 **Reading metrics** — look for the exact moment the metric changed and what changed with it. Not just the current value but the rate of change. ```bash ## Prometheus query: when did error rate start climbing? ## This shows error rate over the last hour with 1-minute resolution ## Look for the exact minute the line starts going up ## In Prometheus UI or Grafana: ## rate(http_requests_total{service="payment-api",status=~"5.."}[1m]) ## Check current database connection count vs maximum ## Shows if connections are at or near the limit ## pg_stat_activity is a PostgreSQL system view showing active connections ## pg_settings shows configuration values like max_connections SELECT count(*) as current, (SELECT setting::int FROM pg_settings WHERE name='max_connections') as maximum FROM pg_stat_activity; ``` **Reading logs** — look for the first error, not the most recent one. The first error is closest to the cause. ```bash ## Get logs from the payment service — look at FIRST errors, not latest ## --since-time gets logs from a specific timestamp ## Adjust the timestamp to just before the incident started kubectl logs -n payment deployment/payment-api --since-time="2024-11-14T14:28:00Z" | head -50 ## Look for connection errors, timeouts, or pool exhaustion specifically kubectl logs -n payment deployment/payment-api --since-time="2024-11-14T14:28:00Z" \ | grep -i "error\|timeout\|exhausted\|refused\|failed" | head -20 ## If pod restarted, get logs from the crashed container ## --previous gets logs from the container instance before the current one kubectl logs -n payment payment-api-7d9b4c8f6-xkp2m --previous | tail -30 ``` **Reading traces** — if you have distributed tracing (covered in the OpenTelemetry module), traces show you exactly which service call is slow or failing in a request chain. ```text Request: POST /payments/process (8200ms total) ├── auth-service.validate_token (45ms) ✓ ├── inventory.check_availability (38ms) ✓ ├── redis-cache.get_session (7890ms) ← THIS IS THE SLOW ONE └── payment-gateway.charge (never reached) ``` The trace tells you instantly: the slowness is in the Redis cache call. Everything before it was fine. Everything after it never happened because the request timed out waiting for Redis. ### Step 4 — Form and test hypotheses After reading the signals, you should have 2-3 possible causes. Do not just pick one and try to fix it. Write them down in order of likelihood and test each with one specific check before attempting a fix. Template for a hypothesis: ```text Hypothesis: Redis connection pool was reduced in the deployment Evidence for: First error appeared 4 minutes after deployment Logs show "connection pool exhausted" repeatedly Traces show Redis calls taking 7+ seconds Test: kubectl describe configmap redis-config -n payment Check if pool_size changed in the last deployment Expected result if correct: pool_size will be 10, down from 100 ``` Running the test: ```bash ## Check the current Redis connection pool config kubectl describe configmap redis-config -n payment | grep pool ## Check what it was before the deployment — get the previous revision kubectl rollout history deployment/payment-api -n payment kubectl rollout history deployment/payment-api -n payment --revision=2 ## Compare the current config with the previous one kubectl get configmap redis-config -n payment -o yaml ``` If the hypothesis is confirmed — pool_size is 10 and it used to be 100 — you have your root cause. You can fix it with confidence. If the hypothesis is not confirmed — pool_size is still 100 — eliminate it and move to your next hypothesis. Do not waste time trying to fix something you have not confirmed. > 🔴 **Common Mistake:** Trying to fix the most likely cause without confirming it first. This leads to making unnecessary changes during an active incident, which can introduce new problems and make the situation worse. Always confirm before fixing. ### Step 5 — Confirm root cause and fix Once a hypothesis is confirmed, apply the fix and watch the signals recover — do not just apply the fix and assume it worked. ```bash ## Fix: restore the Redis connection pool size ## Edit the configmap to restore pool_size to 100 kubectl edit configmap redis-config -n payment ## Restart the deployment to pick up the config change ## rollout restart creates new pods with the updated config kubectl rollout restart deployment/payment-api -n payment ## Watch the rollout complete — do not close this until it finishes kubectl rollout status deployment/payment-api -n payment ## Watch error rate drop in real time ## This command watches pod status every 2 seconds watch kubectl get pods -n payment ``` While the rollout runs, keep your monitoring dashboard open. You should see: * Error rate start dropping within 60-90 seconds * Latency returning to baseline * Redis connection count dropping back to normal If error rate does not drop after the fix is deployed, your hypothesis was wrong or incomplete. Go back to step 4 with new information. > 📌 **Remember:** A successful fix is confirmed by metrics recovering, not just by the absence of new error logs. Always check your monitoring dashboard after applying a fix. "No new errors" is not the same as "the system is healthy again." ---
### What each signal type tells you The three pillars of observability each answer a different question: * **Metrics** answer: is something wrong right now? They give you numbers — error rate, latency, throughput, resource usage. Metrics tell you the what and the when but not the why. * **Logs** answer: what happened? They give you the sequence of events — requests received, database calls made, errors thrown, decisions taken. Logs tell you the story but you have to find the relevant chapter. * **Traces** answer: where exactly did it break? In a distributed system where a single user request touches 5-10 services, traces show you the exact path of that request and exactly which step was slow or failed. None of these alone gives you the full picture. Used together, they triangulate the root cause. ### The most important Prometheus queries for incident response ```bash ## Query 1: Error rate by service — which services are broken? ## rate() calculates per-second average over the time window ## 5.. matches any 5xx status code (500, 502, 503, 504...) sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) ## Query 2: Latency percentiles — how slow is it? ## histogram_quantile calculates the Nth percentile from a histogram metric ## 0.99 = 99th percentile (p99) — the slowest 1% of requests histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) ## Query 3: Is this getting worse or better? ## Compare current error rate to 1 hour ago ## If the result is > 1, it is worse than an hour ago sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total{status=~"5.."}[5m] offset 1h)) by (service) ## Query 4: Resource usage — is anything running out? ## Shows memory usage as a percentage of the limit ## 1 = 100% of memory limit — pod is about to OOMKill container_memory_usage_bytes{namespace="payment"} / container_spec_memory_limit_bytes{namespace="payment"} ## Query 5: Database connection saturation ## pg_stat_activity counts active connections ## pg_settings shows the configured maximum ## If this ratio is close to 1, you are near the connection limit pg_stat_activity_count / pg_settings{name="max_connections"} ``` ### The most useful kubectl commands during an incident ```bash ## Get a complete picture of a failing pod ## describe shows events, resource usage, restart history, and config kubectl describe pod <pod-name> -n <namespace> ## Get logs with timestamps — critical for building a timeline ## --timestamps adds the exact time to every log line kubectl logs <pod-name> -n <namespace> --timestamps=true | tail -50 ## Stream live logs from all pods in a deployment ## -l selects pods by label, -f follows (streams in real time) kubectl logs -n payment -l app=payment-api -f ## Check resource usage right now — is any pod near its limit? ## Shows CPU and memory for every pod kubectl top pods -n payment ## Check node resource usage — is the node itself overwhelmed? kubectl top nodes ## Get the exact reason a pod crashed ## Shows the last exit code and reason kubectl get pod <pod-name> -n payment -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' ## Check if there are any pending pods that cannot be scheduled ## Pending pods often indicate resource exhaustion at the node level kubectl get pods --all-namespaces | grep Pending ``` ### Structured log searching Random grep is not enough during a fast-moving incident. Here is a structured approach: ```bash ## Step 1: Find the first error — not the latest ## Sort logs by timestamp and show the first 20 errors kubectl logs -n payment deployment/payment-api --since-time="2024-11-14T14:28:00Z" \ --timestamps=true | grep -i "error\|exception\|fatal" | head -20 ## Step 2: Find the most common error type ## Count error occurrences by type to identify the dominant failure mode kubectl logs -n payment deployment/payment-api --since-time="2024-11-14T14:28:00Z" \ | grep -i "error" \ | grep -oP '"message":"[^"]*"' \ | sort | uniq -c | sort -rn | head -10 ## Step 3: Check all pods, not just one ## Errors from one pod may differ from errors on another ## Compare to find if the issue is isolated to specific pods for pod in $(kubectl get pods -n payment -l app=payment-api -o name); do echo "=== $pod ===" kubectl logs -n payment $pod --since="10m" | grep -i error | tail -5 done ``` ---
### What we are building A command-line tool that automates the most time-consuming parts of step 3 — pulling logs, metrics, and recent deployment events simultaneously and giving you a structured view of the incident data. Instead of running 10 commands manually, you run one. This is also the foundation of an AIOps RCA agent — once you have this data collected, you can pass it to the LLM prompt templates from the RCA Prompt Templates module to get a structured diagnosis. ### Step 1: Install dependencies ```bash ## Install the Prometheus API client and Kubernetes client pip install prometheus-api-client kubernetes requests python-dotenv ``` ### Step 2: Build the data collector ```python # rca_assistant.py # Collects logs, metrics, and events from a Kubernetes cluster # and formats them into a structured incident report import subprocess import json import sys from datetime import datetime, timedelta from typing import Optional # kubernetes client for cluster API calls from kubernetes import client, config def setup_k8s_client(): """ Load Kubernetes config from the default kubeconfig file (~/.kube/config). This is the same config kubectl uses — if kubectl works, this works. """ try: config.load_kube_config() return client.CoreV1Api(), client.AppsV1Api() except Exception as e: print(f"Could not connect to Kubernetes: {e}") print("Make sure kubectl is configured and the cluster is reachable") sys.exit(1) def get_recent_deployments(apps_v1, namespace: str, hours: int = 2) -> list: """ Get all deployments in the namespace and their recent rollout history. This answers Step 1: what changed before the incident? Returns a list of deployment names and their current image versions. """ deployments = [] try: # List all deployments in the namespace result = apps_v1.list_namespaced_deployment(namespace) for dep in result.items: # Get the container images — these change during deployments containers = dep.spec.template.spec.containers images = [f"{c.name}:{c.image.split(':')[-1]}" for c in containers] deployments.append({ "name": dep.metadata.name, # creation_timestamp is when the current version was deployed "last_updated": dep.metadata.creation_timestamp.strftime("%Y-%m-%d %H:%M:%S"), "replicas_desired": dep.spec.replicas, "replicas_ready": dep.status.ready_replicas or 0, "images": images }) except Exception as e: print(f"Warning: Could not get deployments: {e}") return deployments def get_pod_status(core_v1, namespace: str) -> list: """ Get status of all pods including restart counts and current state. High restart counts are a strong signal of a crash loop or bad config. """ pods = [] try: result = core_v1.list_namespaced_pod(namespace) for pod in result.items: # Get restart count from the first container # A pod with 5+ restarts in the last hour needs investigation restart_count = 0 if pod.status.container_statuses: restart_count = pod.status.container_statuses[0].restart_count pods.append({ "name": pod.metadata.name, "phase": pod.status.phase, # Running, Pending, Failed, etc. "restarts": restart_count, "node": pod.spec.node_name, "ready": all( cs.ready for cs in (pod.status.container_statuses or []) ) }) except Exception as e: print(f"Warning: Could not get pod status: {e}") return pods def get_recent_logs(namespace: str, service: str, since_minutes: int = 15) -> str: """ Get recent logs from all pods of a service using kubectl. Filters for errors only to reduce noise. Uses subprocess to run kubectl — same as running it in the terminal. """ try: # Get logs from all pods with this label, filtered to errors only result = subprocess.run([ "kubectl", "logs", "-n", namespace, "-l", f"app={service}", # select all pods for this service "--since", f"{since_minutes}m", "--timestamps=true" ], capture_output=True, text=True, timeout=30) logs = result.stdout # Filter to only error lines to reduce noise # In a real system you would use a proper log query API error_lines = [ line for line in logs.split('\n') if any(word in line.lower() for word in ['error', 'exception', 'fatal', 'timeout', 'refused']) ] return '\n'.join(error_lines[:30]) if error_lines else "No error logs found" except subprocess.TimeoutExpired: return "Log collection timed out — cluster may be slow" except Exception as e: return f"Could not collect logs: {e}" def get_prometheus_metrics(prometheus_url: str, service: str) -> dict: """ Query Prometheus for the current error rate and latency of the service. Returns a dict with the key health indicators. prometheus_url: base URL of your Prometheus server, e.g. http://localhost:9090 """ import requests metrics = {} try: # Query 1: Current error rate (errors per second) error_query = f'sum(rate(http_requests_total{{service="{service}",status=~"5.."}}[5m]))' resp = requests.get(f"{prometheus_url}/api/v1/query", params={"query": error_query}, timeout=10) data = resp.json() if data["data"]["result"]: # result[0]["value"][1] is the current metric value as a string metrics["error_rate_per_sec"] = float(data["data"]["result"][0]["value"][1]) else: metrics["error_rate_per_sec"] = 0.0 # Query 2: p99 latency latency_query = f'histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{{service="{service}"}}[5m])) by (le))' resp = requests.get(f"{prometheus_url}/api/v1/query", params={"query": latency_query}, timeout=10) data = resp.json() if data["data"]["result"]: metrics["p99_latency_seconds"] = float(data["data"]["result"][0]["value"][1]) else: metrics["p99_latency_seconds"] = 0.0 except Exception as e: metrics["error"] = f"Could not reach Prometheus: {e}" return metrics def build_incident_report(namespace: str, service: str, prometheus_url: Optional[str] = None) -> dict: """ Pull all three signals together into one structured incident report. This is the data you would feed into the RCA prompt template. """ print(f"Collecting incident data for {service} in {namespace}...") core_v1, apps_v1 = setup_k8s_client() report = { "service": service, "namespace": namespace, "collected_at": datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S UTC"), # Step 1: What changed? "recent_deployments": get_recent_deployments(apps_v1, namespace), # Step 2 & 3: What is the blast radius and pod health? "pod_status": get_pod_status(core_v1, namespace), # Step 3: What do the logs say? "recent_error_logs": get_recent_logs(namespace, service), # Step 3: What do the metrics say? "metrics": get_prometheus_metrics(prometheus_url, service) if prometheus_url else {"note": "Prometheus URL not provided"} } return report def print_report(report: dict): """Print the incident report in a readable format for the on-call engineer.""" print("\n" + "="*60) print(f"INCIDENT REPORT: {report['service']} ({report['namespace']})") print(f"Collected at: {report['collected_at']}") print("="*60) print("\n-- RECENT DEPLOYMENTS --") for dep in report["recent_deployments"]: ready = dep["replicas_ready"] desired = dep["replicas_desired"] status = "OK" if ready == desired else f"DEGRADED ({ready}/{desired} ready)" print(f" {dep['name']}: {status} | Last updated: {dep['last_updated']}") for img in dep["images"]: print(f" Image: {img}") print("\n-- POD STATUS --") for pod in report["pod_status"]: flag = "WARN" if pod["restarts"] > 3 or not pod["ready"] else "OK" print(f" [{flag}] {pod['name']} | Phase: {pod['phase']} | Restarts: {pod['restarts']}") print("\n-- METRICS --") metrics = report["metrics"] if "error" not in metrics: print(f" Error rate: {metrics.get('error_rate_per_sec', 0):.3f} errors/sec") print(f" p99 Latency: {metrics.get('p99_latency_seconds', 0)*1000:.0f}ms") else: print(f" {metrics['error']}") print("\n-- RECENT ERROR LOGS --") print(report["recent_error_logs"]) print("="*60) # Usage if __name__ == "__main__": # Collect and display incident report for the payment service report = build_incident_report( namespace="payment", service="payment-api", prometheus_url="http://localhost:9090" # set to your Prometheus URL ) print_report(report) # Optionally save as JSON to feed into the RCA prompt template with open("incident_report.json", "w") as f: json.dump(report, f, indent=2) print("\nFull report saved to incident_report.json") ``` ### Step 3: Run it during an incident ```bash ## Run the RCA assistant for the payment service python3 rca_assistant.py ## The output gives you the structured view immediately: ## - Which deployments happened recently and their status ## - Which pods are unhealthy or restarting ## - Current error rate and latency from Prometheus ## - Recent error logs filtered from noise ``` ### Step 4: Feed the report into an LLM for diagnosis Once you have the incident report, pass it directly into the RCA prompt template: ```python from openai import OpenAI import json client = OpenAI() # Load the collected incident data with open("incident_report.json") as f: report = json.load(f) # Build the RCA prompt using the collected data prompt = f"""You are a senior SRE. Analyze this incident data and identify the root cause. SERVICE: {report['service']} NAMESPACE: {report['namespace']} COLLECTED AT: {report['collected_at']} RECENT DEPLOYMENTS: {json.dumps(report['recent_deployments'], indent=2)} POD STATUS: {json.dumps(report['pod_status'], indent=2)} METRICS: {json.dumps(report['metrics'], indent=2)} ERROR LOGS: {report['recent_error_logs']} Respond in this format: - Root Cause: one sentence - Confidence: HIGH / MEDIUM / LOW - Evidence: 2-3 bullet points from the data above - Next 3 Diagnostic Steps: numbered list - Escalate if: one sentence""" response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], temperature=0.1 # low temperature for consistent, factual analysis ) print(response.choices[0].message.content) ``` ---
### What a postmortem is for A postmortem is not a blame document. It is not a summary of what happened. It is an analysis of why the system allowed this failure to happen, what made it possible, and what specific changes will prevent it recurring. The difference: ```text Bad postmortem (what happened): "The payment service went down because a deployment changed the Redis connection pool size from 100 to 10. The engineer fixed it by changing it back. Duration: 22 minutes." Good postmortem (why it was possible): "A configuration change reduced Redis pool size from 100 to 10. This change was not caught in review because: 1. No automated test validates Redis pool configuration 2. The config change was in a separate PR from the code change 3. Staging environment uses a pool size of 5, masking the problem Action items: 1. Add automated test that validates pool_size >= 50 in all environments 2. Add config validation step to deployment pipeline 3. Standardize staging pool size to match production ratio" ``` The good postmortem identifies three systemic gaps. Fixing all three means this specific failure pattern cannot recur. ### The postmortem structure ```text ## Incident Summary One paragraph: what happened, when, duration, user impact. ## Timeline Exact timestamps of every significant event. Format: HH:MM UTC — what happened ## Root Cause One sentence: the specific underlying cause. Not "Redis was slow" but "Redis connection pool reduced from 100 to 10 in deployment payment-api v2.4.1 at 14:28 UTC" ## Contributing Factors 2-5 factors that made this failure possible. Each one is a systemic gap, not a human error. ## Impact Number of users affected, revenue impact if known, duration. ## What Went Well 2-3 things that worked correctly during the incident. This is not empty positivity — it identifies what to preserve. ## Action Items Table format: item, owner, due date, status. Every item must be specific and testable. "Improve monitoring" is not an action item. "Add alert for Redis pool_size < 50 in Prometheus" is. ``` ---
### RCA command reference | Command | What it tells you | |---------|------------------| | `kubectl get events -n <ns> --sort-by='.lastTimestamp'` | Recent cluster events in time order | | `kubectl rollout history deployment/<name> -n <ns>` | Recent deployment history | | `kubectl logs <pod> -n <ns> --previous` | Logs from crashed container | | `kubectl logs -n <ns> -l app=<service> --since=15m` | Logs from all pods of a service | | `kubectl describe pod <pod> -n <ns>` | Full pod details including events and resource usage | | `kubectl top pods -n <ns>` | Current CPU and memory usage per pod | | `kubectl get pods --all-namespaces \| grep -v Running` | All unhealthy pods across the cluster | ### Common mistakes in production debugging **Starting with a fix instead of a diagnosis.** The most common mistake. An engineer sees high error rate, assumes it is a memory issue, restarts pods. Errors come back. They restart again. The real cause — a bad config value — is never found. Always spend 3-5 minutes on steps 1-3 before touching anything. **Reading logs from the wrong time range.** Engineers often check the most recent logs. But if the incident started 20 minutes ago, the cause is in logs from 20 minutes ago — not the last 5 minutes. Always set `--since-time` to just before the incident started, not a fixed window. **Checking only one pod.** If a service has 3 pods and one is bad, error rate will be around 33%. Checking only the healthy pods finds nothing. Always check logs across all pods for a service, not just one. **Confirming a fix with absence of errors instead of metric recovery.** "I don't see any more errors" is not confirmation. Check your monitoring dashboard and confirm error rate has returned to baseline. A fix that suppresses errors without fixing the underlying cause will fail again. **Postmortems that blame individuals.** A postmortem that says "the engineer deployed without proper testing" identifies a person, not a systemic gap. The same mistake will happen again with a different engineer. Always ask: what in the system allowed this to happen? What process or automation was missing? **Not updating the runbook after the incident.** After every incident where the runbook was wrong, missing, or incomplete — update it immediately while the details are fresh. The next on-call engineer should not have to rediscover what you just learned.
The problem with guessing during incidents It is 3 AM. An alert fires. Error rate on the payment service jumped from 0.1...
Why a fixed process matters under pressure When you are under pressure at 3 AM, cognitive load is high. A fixed process ...
What each signal type tells you The three pillars of observability each answer a different question: Metrics answer: is ...
What we are building A command-line tool that automates the most time-consuming parts of step 3 — pulling logs, metrics,...
What a postmortem is for A postmortem is not a blame document. It is not a summary of what happened. It is an analysis o...
RCA command reference Command What it tells you kubectl get events -n <ns> --sort-by='.lastTimestamp' Recent cluster eve...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.