project - Build an AI-Powered Incident Response Agent | Engineering Modules | DevOps Network

Engineering ModuleINTERMEDIATE

project - Build an AI-Powered Incident Response Agent

Build an AI agent that receives production alerts, queries Prometheus for context, retrieves runbooks via RAG, and suggests remediation steps automatically.

~240 Minutes

16 Topics

Hands-on Scenarios

What You'll Learn

Before You Start — Read This First

### What problem are we actually solving It is 3 AM. Your phone rings — PagerDuty. The `payment-service` is throwing 500 errors. You open your laptop, half asleep. You need to: 1. Find out which pods are affected 2. Check CPU and memory in Prometheus 3. Look up the runbook for this alert 4. Decide whether to restart the pod, scale up, or escalate That whole process takes 15 to 30 minutes. You repeat it dozens of times a month. Each time you are slow, customers cannot pay. At Razorpay, every minute of payment downtime is measured in crores of lost transactions. This capstone builds the system that does steps 1 through 4 automatically — before you even look at your laptop. The agent receives an alert, collects context from your live infrastructure, searches your runbook library, and replies with a structured diagnosis and remediation plan. You still make the final call. But instead of a blank terminal at 3 AM, you get a clear answer with everything assembled for you. **What you will build:** * A vector database loaded with all your runbooks and past incident reports * A Python agent that receives Prometheus/Alertmanager webhooks * A context collector that pulls live metrics for the affected service * A RAG (Retrieval-Augmented Generation) pipeline that finds relevant runbooks * An LLM reasoning step that synthesises everything into a structured diagnosis * A Slack notifier that posts the result to your on-call channel Time to complete: 4-5 hours. **What you need before starting:** * Completed Capstone 1 (Anomaly Detection Pipeline) — the Prometheus setup is the same * Python 3.9+ with the Capstone 1 dependencies installed * Ollama running locally (from the Model Ecosystem module) * A running Kubernetes cluster with Prometheus and Alertmanager ```bash ## Verify Ollama is installed and a model is available ollama list ## Should show at least one model (llama3.1 or mistral) ## If no model is downloaded: ollama pull llama3.1 ## ~4GB — do this before starting ## Verify Prometheus is accessible curl -s http://localhost:9090/api/v1/query?query=up | python3 -m json.tool | head -5 ## Should show JSON with "status": "success" echo "✅ Ready to build" ```

Understanding the Architecture

### How the three layers connect This agent has three layers working in sequence. Understanding how they connect before writing any code will make every piece of the implementation obvious. ┌─────────────────────────────────────────────────────┐ │ LAYER 1: TRIGGER │ │ Alertmanager fires webhook → FastAPI endpoint │ │ receives JSON alert payload │ └──────────────────────┬──────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ LAYER 2: CONTEXT │ │ Prometheus queries → live CPU, memory, error rate │ │ kubectl events → recent crash/OOM events │ │ Alert metadata → service name, severity, labels │ └──────────────────────┬──────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ LAYER 3: REASONING │ │ RAG search → finds relevant runbooks from │ │ ChromaDB vector database │ │ LLM prompt → combines context + runbooks │ │ → structured diagnosis + remediation steps │ └──────────────────────┬──────────────────────────────┘ │ ▼ Slack message to #on-call channel **Layer 1** (Trigger) is just a webhook receiver. Alertmanager already knows how to send webhooks — you just need an endpoint to receive them. **Layer 2** (Context) is the most important layer. Without real context about the current state of the system, the LLM will give generic advice. With context, it gives specific, actionable diagnosis. **Layer 3** (Reasoning) is where RAG and the LLM work together. RAG finds the right runbook. The LLM reads the runbook plus the live context and generates a diagnosis that is specific to this exact incident. ### Why RAG is necessary here Without RAG, you would send your entire runbook library in every prompt. That is thousands of lines of text — too expensive, too slow, and the LLM gets confused by irrelevant content. RAG solves this by converting runbooks into vectors (numerical representations of meaning) and storing them in ChromaDB. When an alert arrives, the alert description is converted to a vector and compared against all runbook vectors. The 3 most relevant runbooks are returned in milliseconds. Only those 3 are included in the LLM prompt. Think of it like a search engine for meaning, not keywords. If your alert says "pod OOMKilled" and your runbook says "memory limit exceeded causing container restart", RAG finds it — even though none of the words match exactly.

Writing Sample Runbooks

### Why runbooks must exist before the agent can work The agent is only as useful as the runbook library behind it. Before building the agent code, write the runbooks it will retrieve. In real teams these are maintained in Confluence, Notion, or Git. For this capstone you will create three realistic runbooks as markdown files. ```bash ## Create the first runbook cat > runbooks/oom-kill-runbook.md << 'EOF' # OOMKill Runbook ## Symptoms Pod is in CrashLoopBackOff state. kubectl describe pod shows Reason: OOMKilled. Memory usage was at or near the container memory limit immediately before crash. ## Root Causes 1. Memory limit set too low for the workload 2. Memory leak in application code (usage grows continuously) 3. Sudden traffic spike causing higher memory usage than baseline ## Immediate Remediation Steps ### Step 1: Identify affected pods kubectl get pods -n <namespace> | grep CrashLoopBackOff ### Step 2: Check recent events kubectl describe pod <pod-name> -n <namespace> | tail -20 ### Step 3: Temporary fix - increase memory limit kubectl patch deployment <deployment-name> -n <namespace> \ --patch '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"limits":{"memory":"512Mi"}}}]}}}}' ### Step 4: Check if memory usage is growing (leak indicator) kubectl top pod <pod-name> -n <namespace> ## If memory grows steadily after restart, escalate to dev team - likely a leak ## Escalation Criteria Escalate to on-call engineering lead if: - Pod restarts more than 5 times in 10 minutes - Increasing memory limit does not stop the crashes - Multiple services affected simultaneously ## Prevention Set memory requests equal to 70% of memory limits to give headroom. Add memory usage alerts at 80% of limit. EOF echo "✅ OOMKill runbook created" ``` ```bash ## Create the second runbook cat > runbooks/high-error-rate-runbook.md << 'EOF' # High HTTP Error Rate Runbook ## Symptoms HTTP 5xx error rate exceeds 5% for a sustained period (more than 2 minutes). Users report failures or timeouts. Dashboards show error rate spike. ## Root Causes 1. Downstream dependency (database, cache, external API) is unavailable 2. Recent deployment introduced a bug 3. Pod is running out of resources (CPU throttling causing timeouts) 4. Connection pool exhausted ## Immediate Remediation Steps ### Step 1: Check if this started after a deployment kubectl rollout history deployment/<deployment-name> -n <namespace> ## Look for a recent rollout timestamp matching the error spike ### Step 2: If recent deployment - rollback immediately kubectl rollout undo deployment/<deployment-name> -n <namespace> kubectl rollout status deployment/<deployment-name> -n <namespace> ### Step 3: Check downstream dependencies kubectl get pods -n <namespace> ## Look for any pods in Error or CrashLoopBackOff state ### Step 4: Check CPU throttling kubectl top pods -n <namespace> ## High CPU usage + 5xx errors often means CPU throttling causing timeouts ### Step 5: Check application logs for the actual error kubectl logs deployment/<deployment-name> -n <namespace> --tail=50 | grep -i error ## Escalation Criteria Escalate if rollback does not reduce error rate within 5 minutes. Escalate if database or cache pods are crashing. ## Prevention Deploy with canary releases (10% traffic first). Set CPU requests accurately to prevent throttling. Add downstream health checks. EOF echo "✅ High error rate runbook created" ``` ```bash ## Create the third runbook cat > runbooks/pod-crash-loop-runbook.md << 'EOF' # CrashLoopBackOff Runbook ## Symptoms Pod status shows CrashLoopBackOff. kubectl logs shows process exiting immediately. Restart count is increasing. The pod never reaches Running state successfully. ## Root Causes 1. Application exits on startup due to missing environment variable or config 2. Liveness probe failing immediately after start 3. Image pull error or wrong image tag deployed 4. Permission denied error on mounted volume ## Immediate Remediation Steps ### Step 1: Get the crash logs (last run) kubectl logs <pod-name> -n <namespace> --previous ## --previous shows logs from the container that just crashed ### Step 2: Check the exit reason kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Last State" ## Exit Code 1 = application error, Exit Code 137 = OOMKill, Exit Code 0 = clean exit ### Step 3: Check environment variables kubectl exec -it <pod-name> -n <namespace> -- env | grep -i required_var ## If the pod starts briefly before crashing, this shows if config is missing ### Step 4: Check if image exists and is correct kubectl describe pod <pod-name> -n <namespace> | grep Image ## Verify the image tag matches what was deployed ### Step 5: Check volume mounts kubectl describe pod <pod-name> -n <namespace> | grep -A10 Volumes ## Permission denied on mounted secrets or configmaps causes immediate crash ## Escalation Criteria If exit code is 0 (clean exit on start) and the application seems correct - the liveness probe configuration is likely wrong. Escalate to the team that owns the service to review probe timing. ## Prevention Use init containers to validate config before the main container starts. Set initialDelaySeconds on liveness probes to at least 30 seconds. EOF echo "✅ CrashLoopBackOff runbook created" ## Verify all runbooks are present ls -la runbooks/ ```

Building the Configuration

### All settings in one file ```python # src/config.py import os # ── Prometheus ─────────────────────────────────────────────── PROMETHEUS_URL = os.getenv("PROMETHEUS_URL", "http://localhost:9090") # ── ChromaDB ───────────────────────────────────────────────── # ChromaDB stores vectors on disk in this directory # Persists between restarts so you do not re-index on every run CHROMA_DB_PATH = "data/chroma" CHROMA_COLLECTION_NAME = "runbooks" # ── Embedding Model ────────────────────────────────────────── # This model converts text to vectors # all-MiniLM-L6-v2 is small (90MB), fast, and good enough for ops text EMBEDDING_MODEL = "all-MiniLM-L6-v2" # ── LLM (via Ollama) ───────────────────────────────────────── OLLAMA_URL = os.getenv("OLLAMA_URL", "http://localhost:11434") # Change to "mistral" if you pulled that model instead LLM_MODEL = os.getenv("LLM_MODEL", "llama3.1") # ── Slack ───────────────────────────────────────────────────── # Set this environment variable with your Slack webhook URL # Get it from: Slack → Apps → Incoming Webhooks → Add New Webhook SLACK_WEBHOOK_URL = os.getenv("SLACK_WEBHOOK_URL", "") # ── Webhook Receiver ───────────────────────────────────────── WEBHOOK_HOST = "0.0.0.0" WEBHOOK_PORT = int(os.getenv("WEBHOOK_PORT", "8080")) # ── RAG Settings ───────────────────────────────────────────── # How many runbook chunks to retrieve per alert RAG_TOP_K = 3 # ── Logging ────────────────────────────────────────────────── LOG_PATH = "logs/agent.log" ```

Building the Runbook Indexer

### How indexing works Before the agent can search runbooks, it must convert them from text into vectors and store them in ChromaDB. This is called **indexing**, and you do it once. After that, ChromaDB keeps the vectors on disk and the agent searches them at runtime without re-indexing. Each runbook is split into **chunks** — small paragraphs of about 200-300 words. This matters because you want to retrieve the specific section of a runbook that is relevant, not the entire document. A runbook about OOMKills has a "symptoms" section, a "remediation" section, and an "escalation" section. If your alert is about symptoms, you want the symptoms section, not the escalation section. ```python # src/runbook_indexer.py # Loads runbook files from the runbooks/ directory, # splits them into chunks, and indexes them in ChromaDB. import os import glob import chromadb from sentence_transformers import SentenceTransformer from src.config import CHROMA_DB_PATH, CHROMA_COLLECTION_NAME, EMBEDDING_MODEL def chunk_text(text, chunk_size=300, overlap=50): """ Split text into overlapping chunks of approximately chunk_size words. Overlap ensures that context at the boundary between chunks is not lost. For example, if a runbook step spans the end of chunk 1 and the start of chunk 2, the overlap means it appears in both chunks and will be retrieved by either one. """ words = text.split() chunks = [] start = 0 while start < len(words): end = start + chunk_size chunk = " ".join(words[start:end]) chunks.append(chunk) start += chunk_size - overlap ## move forward by chunk_size minus overlap return chunks def build_runbook_index(): """ Index all markdown files in the runbooks/ directory into ChromaDB. Safe to run multiple times -- existing entries are deleted and re-indexed. """ ## Connect to ChromaDB (creates the directory if it does not exist) client = chromadb.PersistentClient(path=CHROMA_DB_PATH) ## Delete and recreate the collection for a clean index ## This makes re-indexing safe if runbooks are updated try: client.delete_collection(CHROMA_COLLECTION_NAME) print(f" Deleted existing collection '{CHROMA_COLLECTION_NAME}'") except Exception: pass ## collection did not exist yet collection = client.create_collection( name=CHROMA_COLLECTION_NAME, ## Cosine similarity is better than Euclidean distance for text ## It measures the angle between vectors, ignoring magnitude metadata={"hnsw:space": "cosine"}, ) ## Load the embedding model print(f"📦 Loading embedding model '{EMBEDDING_MODEL}'...") print(f" (First run downloads ~90MB -- subsequent runs are instant)") embedder = SentenceTransformer(EMBEDDING_MODEL) ## Find all runbook markdown files runbook_files = glob.glob("runbooks/*.md") if not runbook_files: print("❌ No runbooks found in runbooks/ directory") print(" Create .md files in that directory first") return False print(f"\n📚 Indexing {len(runbook_files)} runbooks...") total_chunks = 0 all_ids = [] all_texts = [] all_embeddings = [] all_metadatas = [] for filepath in runbook_files: filename = os.path.basename(filepath) print(f" Processing {filename}...", end=" ") with open(filepath, "r") as f: content = f.read() ## Split the runbook into chunks chunks = chunk_text(content) print(f"{len(chunks)} chunks") for i, chunk in enumerate(chunks): chunk_id = f"{filename}_chunk_{i}" all_ids.append(chunk_id) all_texts.append(chunk) all_metadatas.append({ "source_file": filename, "chunk_index": i, "total_chunks": len(chunks), }) total_chunks += len(chunks) ## Encode all chunks at once (faster than one-by-one) print(f"\n🔢 Generating embeddings for {total_chunks} chunks...") all_embeddings = embedder.encode(all_texts, show_progress_bar=True).tolist() ## Add everything to ChromaDB in one batch collection.add( ids=all_ids, documents=all_texts, embeddings=all_embeddings, metadatas=all_metadatas, ) print(f"\n✅ Indexed {total_chunks} chunks from {len(runbook_files)} runbooks") print(f" Stored in: {CHROMA_DB_PATH}") ## Verify the index by running a test search print(f"\n🔍 Test search: 'pod is OOMKilled and crashing'") test_embedding = embedder.encode(["pod is OOMKilled and crashing"]).tolist() results = collection.query(query_embeddings=test_embedding, n_results=1) if results["documents"][0]: print(f" Found: {results['metadatas'][0][0]['source_file']}") print(f" Preview: {results['documents'][0][0][:100]}...") return True if __name__ == "__main__": build_runbook_index() ``` ```bash ## Index the runbooks python3 -m src.runbook_indexer ## Expected output: ## 📦 Loading embedding model 'all-MiniLM-L6-v2'... ## 📚 Indexing 3 runbooks... ## Processing oom-kill-runbook.md... 3 chunks ## Processing high-error-rate-runbook.md... 3 chunks ## Processing pod-crash-loop-runbook.md... 3 chunks ## 🔢 Generating embeddings for 9 chunks... ## ✅ Indexed 9 chunks from 3 runbooks ## 🔍 Test search: 'pod is OOMKilled and crashing' ## Found: oom-kill-runbook.md ## Check what was created ls -lh data/chroma/ ```

project - Build an AI-Powered Incident Response Agent

What You'll Learn

Before You Start — Read This First

Understanding the Architecture

Setting Up the Project

Writing Sample Runbooks

Building the Configuration

Building the Runbook Indexer

Skills You'll Master

Curriculum Index

Before You Start — Read This First

Understanding the Architecture

Setting Up the Project

Writing Sample Runbooks

Building the Configuration

Building the Runbook Indexer

Building the RAG Retriever

Building the Context Collector

Building the Agent Core

Building the Webhook Receiver

Building the Slack Notifier

Running and Testing the Agent

Deploy to Kubernetes

Production Checklist

Common Production Mistakes

Quick Reference

Career Impact

Troubleshooting & Lab Guides

How to Troubleshoot a Linux Production Server: A Systematic Approach

Next Modules

Terraform

Azure DevOps

GitLab

Related Guides

What Is AI SRE? How AI Agents Are Changing Incident Response

GitOps Showdown: ArgoCD vs FluxCD for Kubernetes Teams

Platform Engineering 101: Building an Internal Developer Platform with Backstage