Build an AI-Powered Incident Response Agent

Build an AI agent that receives production alerts, queries Prometheus for context, retrieves runbooks via RAG, and suggests remediation steps automatically.

Related Concepts & TermsPrometheus

Domains & Technologies

Domains

CAPSTONEAIOPSAI-AGENTRAGPAGERDUTY

Technologies

PROMETHEUSPYTHONKUBERNETES

Blueprint Walkthrough

Before You Start — Read This First

What problem are we actually solving

It is 3 AM. Your phone rings — PagerDuty. The payment-service is throwing 500 errors.

You open your laptop, half asleep. You need to:

Find out which pods are affected
Check CPU and memory in Prometheus
Look up the runbook for this alert
Decide whether to restart the pod, scale up, or escalate

That whole process takes 15 to 30 minutes. You repeat it dozens of times a month. Each time you are slow, customers cannot pay. At Razorpay, every minute of payment downtime is measured in crores of lost transactions.

This capstone builds the system that does steps 1 through 4 automatically — before you even look at your laptop.

The agent receives an alert, collects context from your live infrastructure, searches your runbook library, and replies with a structured diagnosis and remediation plan. You still make the final call. But instead of a blank terminal at 3 AM, you get a clear answer with everything assembled for you.

What you will build:

A vector database loaded with all your runbooks and past incident reports
A Python agent that receives Prometheus/Alertmanager webhooks
A context collector that pulls live metrics for the affected service
A RAG (Retrieval-Augmented Generation) pipeline that finds relevant runbooks
An LLM reasoning step that synthesises everything into a structured diagnosis
A Slack notifier that posts the result to your on-call channel

Time to complete: 4-5 hours.

What you need before starting:

Completed Capstone 1 (Anomaly Detection Pipeline) — the Prometheus setup is the same
Python 3.9+ with the Capstone 1 dependencies installed
Ollama running locally (from the Model Ecosystem module)
A running Kubernetes cluster with Prometheus and Alertmanager

Bash

## Verify Ollama is installed and a model is available
ollama list
## Should show at least one model (llama3.1 or mistral)
 
## If no model is downloaded:
ollama pull llama3.1   ## ~4GB — do this before starting
 
## Verify Prometheus is accessible
curl -s http://localhost:9090/api/v1/query?query=up | python3 -m json.tool | head -5
## Should show JSON with "status": "success"
 
echo "✅ Ready to build"

Understanding the Architecture

How the three layers connect

This agent has three layers working in sequence. Understanding how they connect before writing any code will make every piece of the implementation obvious.

Bash

┌─────────────────────────────────────────────────────┐
│                    LAYER 1: TRIGGER                 │
│  Alertmanager fires webhook → FastAPI endpoint      │
│  receives JSON alert payload                        │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│                  LAYER 2: CONTEXT                   │
│  Prometheus queries → live CPU, memory, error rate  │
│  kubectl events → recent crash/OOM events           │
│  Alert metadata → service name, severity, labels    │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────┐
│                 LAYER 3: REASONING                  │
│  RAG search → finds relevant runbooks from          │
│  ChromaDB vector database                           │
│  LLM prompt → combines context + runbooks           │
│  → structured diagnosis + remediation steps         │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
          Slack message to #on-call channel

Layer 1 (Trigger) is just a webhook receiver. Alertmanager already knows how to send webhooks — you just need an endpoint to receive them.

Layer 2 (Context) is the most important layer. Without real context about the current state of the system, the LLM will give generic advice. With context, it gives specific, actionable diagnosis.

Layer 3 (Reasoning) is where RAG and the LLM work together. RAG finds the right runbook. The LLM reads the runbook plus the live context and generates a diagnosis that is specific to this exact incident.

Why RAG is necessary here

Without RAG, you would send your entire runbook library in every prompt. That is thousands of lines of text — too expensive, too slow, and the LLM gets confused by irrelevant content.

RAG solves this by converting runbooks into vectors (numerical representations of meaning) and storing them in ChromaDB. When an alert arrives, the alert description is converted to a vector and compared against all runbook vectors. The 3 most relevant runbooks are returned in milliseconds. Only those 3 are included in the LLM prompt.

Think of it like a search engine for meaning, not keywords. If your alert says "pod OOMKilled" and your runbook says "memory limit exceeded causing container restart", RAG finds it — even though none of the words match exactly.

Setting Up the Project

Create the directory structure

Bash

mkdir aiops-incident-agent && cd aiops-incident-agent
 
## Create all directories and files
mkdir -p src runbooks data logs k8s
 
touch src/config.py
touch src/webhook_receiver.py
touch src/context_collector.py
touch src/runbook_indexer.py
touch src/rag_retriever.py
touch src/agent.py
touch src/notifier.py
touch requirements.txt
 
echo "✅ Project structure created"

Your project structure:

Bash

aiops-incident-agent/
  src/
    config.py            <- all settings in one place
    webhook_receiver.py  <- FastAPI server to receive alerts
    context_collector.py <- pulls live data from Prometheus + kubectl
    runbook_indexer.py   <- loads runbooks into ChromaDB
    rag_retriever.py     <- searches runbooks by similarity
    agent.py             <- the main reasoning loop
    notifier.py          <- sends results to Slack
  runbooks/              <- your markdown runbook files go here
  data/                  <- ChromaDB persists here
  logs/                  <- agent logs
  k8s/                   <- Kubernetes manifests

Install dependencies

Bash

# requirements.txt
fastapi==0.104.1
uvicorn==0.24.0
chromadb==0.4.18
sentence-transformers==2.2.2
requests==2.31.0
pydantic==2.5.0
python-dotenv==1.0.0

Bash

pip3 install -r requirements.txt
## sentence-transformers downloads a ~90MB model on first use
## This is used to convert text into vectors
 
echo "✅ Dependencies installed"

Writing Sample Runbooks

Why runbooks must exist before the agent can work

The agent is only as useful as the runbook library behind it. Before building the agent code, write the runbooks it will retrieve. In real teams these are maintained in Confluence, Notion, or Git. For this capstone you will create three realistic runbooks as markdown files.

Bash

## Create the first runbook
cat > runbooks/oom-kill-runbook.md << 'EOF'
# OOMKill Runbook
 
## Symptoms
Pod is in CrashLoopBackOff state. kubectl describe pod shows Reason: OOMKilled.
Memory usage was at or near the container memory limit immediately before crash.
 
## Root Causes
1. Memory limit set too low for the workload
2. Memory leak in application code (usage grows continuously)
3. Sudden traffic spike causing higher memory usage than baseline
 
## Immediate Remediation Steps
 
### Step 1: Identify affected pods
kubectl get pods -n <namespace> | grep CrashLoopBackOff
 
### Step 2: Check recent events
kubectl describe pod <pod-name> -n <namespace> | tail -20
 
### Step 3: Temporary fix - increase memory limit
kubectl patch deployment <deployment-name> -n <namespace> \
  --patch '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"limits":{"memory":"512Mi"}}}]}}}}'
 
### Step 4: Check if memory usage is growing (leak indicator)
kubectl top pod <pod-name> -n <namespace>
## If memory grows steadily after restart, escalate to dev team - likely a leak
 
## Escalation Criteria
Escalate to on-call engineering lead if:
- Pod restarts more than 5 times in 10 minutes
- Increasing memory limit does not stop the crashes
- Multiple services affected simultaneously
 
## Prevention
Set memory requests equal to 70% of memory limits to give headroom.
Add memory usage alerts at 80% of limit.
EOF
 
echo "✅ OOMKill runbook created"

Bash

## Create the second runbook
cat > runbooks/high-error-rate-runbook.md << 'EOF'
# High HTTP Error Rate Runbook
 
## Symptoms
HTTP 5xx error rate exceeds 5% for a sustained period (more than 2 minutes).
Users report failures or timeouts. Dashboards show error rate spike.
 
## Root Causes
1. Downstream dependency (database, cache, external API) is unavailable
2. Recent deployment introduced a bug
3. Pod is running out of resources (CPU throttling causing timeouts)
4. Connection pool exhausted
 
## Immediate Remediation Steps
 
### Step 1: Check if this started after a deployment
kubectl rollout history deployment/<deployment-name> -n <namespace>
## Look for a recent rollout timestamp matching the error spike
 
### Step 2: If recent deployment - rollback immediately
kubectl rollout undo deployment/<deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>
 
### Step 3: Check downstream dependencies
kubectl get pods -n <namespace>
## Look for any pods in Error or CrashLoopBackOff state
 
### Step 4: Check CPU throttling
kubectl top pods -n <namespace>
## High CPU usage + 5xx errors often means CPU throttling causing timeouts
 
### Step 5: Check application logs for the actual error
kubectl logs deployment/<deployment-name> -n <namespace> --tail=50 | grep -i error
 
## Escalation Criteria
Escalate if rollback does not reduce error rate within 5 minutes.
Escalate if database or cache pods are crashing.
 
## Prevention
Deploy with canary releases (10% traffic first).
Set CPU requests accurately to prevent throttling.
Add downstream health checks.
EOF
 
echo "✅ High error rate runbook created"

Bash

## Create the third runbook
cat > runbooks/pod-crash-loop-runbook.md << 'EOF'
# CrashLoopBackOff Runbook
 
## Symptoms
Pod status shows CrashLoopBackOff. kubectl logs shows process exiting immediately.
Restart count is increasing. The pod never reaches Running state successfully.
 
## Root Causes
1. Application exits on startup due to missing environment variable or config
2. Liveness probe failing immediately after start
3. Image pull error or wrong image tag deployed
4. Permission denied error on mounted volume
 
## Immediate Remediation Steps
 
### Step 1: Get the crash logs (last run)
kubectl logs <pod-name> -n <namespace> --previous
## --previous shows logs from the container that just crashed
 
### Step 2: Check the exit reason
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Last State"
## Exit Code 1 = application error, Exit Code 137 = OOMKill, Exit Code 0 = clean exit
 
### Step 3: Check environment variables
kubectl exec -it <pod-name> -n <namespace> -- env | grep -i required_var
## If the pod starts briefly before crashing, this shows if config is missing
 
### Step 4: Check if image exists and is correct
kubectl describe pod <pod-name> -n <namespace> | grep Image
## Verify the image tag matches what was deployed
 
### Step 5: Check volume mounts
kubectl describe pod <pod-name> -n <namespace> | grep -A10 Volumes
## Permission denied on mounted secrets or configmaps causes immediate crash
 
## Escalation Criteria
If exit code is 0 (clean exit on start) and the application seems correct - the
liveness probe configuration is likely wrong. Escalate to the team that owns the
service to review probe timing.
 
## Prevention
Use init containers to validate config before the main container starts.
Set initialDelaySeconds on liveness probes to at least 30 seconds.
EOF
 
echo "✅ CrashLoopBackOff runbook created"
 
## Verify all runbooks are present
ls -la runbooks/

Building the Configuration

All settings in one file

PYTHON

# src/config.py
import os
 
# ── Prometheus ───────────────────────────────────────────────
PROMETHEUS_URL = os.getenv("PROMETHEUS_URL", "http://localhost:9090")
 
# ── ChromaDB ─────────────────────────────────────────────────
# ChromaDB stores vectors on disk in this directory
# Persists between restarts so you do not re-index on every run
CHROMA_DB_PATH = "data/chroma"
CHROMA_COLLECTION_NAME = "runbooks"
 
# ── Embedding Model ──────────────────────────────────────────
# This model converts text to vectors
# all-MiniLM-L6-v2 is small (90MB), fast, and good enough for ops text
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
 
# ── LLM (via Ollama) ─────────────────────────────────────────
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://localhost:11434")
# Change to "mistral" if you pulled that model instead
LLM_MODEL = os.getenv("LLM_MODEL", "llama3.1")
 
# ── Slack ─────────────────────────────────────────────────────
# Set this environment variable with your Slack webhook URL
# Get it from: Slack → Apps → Incoming Webhooks → Add New Webhook
SLACK_WEBHOOK_URL = os.getenv("SLACK_WEBHOOK_URL", "")
 
# ── Webhook Receiver ─────────────────────────────────────────
WEBHOOK_HOST = "0.0.0.0"
WEBHOOK_PORT = int(os.getenv("WEBHOOK_PORT", "8080"))
 
# ── RAG Settings ─────────────────────────────────────────────
# How many runbook chunks to retrieve per alert
RAG_TOP_K = 3
 
# ── Logging ──────────────────────────────────────────────────
LOG_PATH = "logs/agent.log"

Building the Runbook Indexer

How indexing works

Before the agent can search runbooks, it must convert them from text into vectors and store them in ChromaDB. This is called indexing, and you do it once. After that, ChromaDB keeps the vectors on disk and the agent searches them at runtime without re-indexing.

Each runbook is split into chunks — small paragraphs of about 200-300 words. This matters because you want to retrieve the specific section of a runbook that is relevant, not the entire document. A runbook about OOMKills has a "symptoms" section, a "remediation" section, and an "escalation" section. If your alert is about symptoms, you want the symptoms section, not the escalation section.

PYTHON

# src/runbook_indexer.py
# Loads runbook files from the runbooks/ directory,
# splits them into chunks, and indexes them in ChromaDB.
 
import os
import glob
import chromadb
from sentence_transformers import SentenceTransformer
from src.config import CHROMA_DB_PATH, CHROMA_COLLECTION_NAME, EMBEDDING_MODEL
 
 
def chunk_text(text, chunk_size=300, overlap=50):
    """
    Split text into overlapping chunks of approximately chunk_size words.
 
    Overlap ensures that context at the boundary between chunks is not lost.
    For example, if a runbook step spans the end of chunk 1 and the start of
    chunk 2, the overlap means it appears in both chunks and will be retrieved
    by either one.
    """
    words = text.split()
    chunks = []
    start = 0
 
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        chunks.append(chunk)
        start += chunk_size - overlap   ## move forward by chunk_size minus overlap
 
    return chunks
 
 
def build_runbook_index():
    """
    Index all markdown files in the runbooks/ directory into ChromaDB.
    Safe to run multiple times -- existing entries are deleted and re-indexed.
    """
 
    ## Connect to ChromaDB (creates the directory if it does not exist)
    client = chromadb.PersistentClient(path=CHROMA_DB_PATH)
 
    ## Delete and recreate the collection for a clean index
    ## This makes re-indexing safe if runbooks are updated
    try:
        client.delete_collection(CHROMA_COLLECTION_NAME)
        print(f"   Deleted existing collection '{CHROMA_COLLECTION_NAME}'")
    except Exception:
        pass   ## collection did not exist yet
 
    collection = client.create_collection(
        name=CHROMA_COLLECTION_NAME,
        ## Cosine similarity is better than Euclidean distance for text
        ## It measures the angle between vectors, ignoring magnitude
        metadata={"hnsw:space": "cosine"},
    )
 
    ## Load the embedding model
    print(f"📦 Loading embedding model '{EMBEDDING_MODEL}'...")
    print(f"   (First run downloads ~90MB -- subsequent runs are instant)")
    embedder = SentenceTransformer(EMBEDDING_MODEL)
 
    ## Find all runbook markdown files
    runbook_files = glob.glob("runbooks/*.md")
 
    if not runbook_files:
        print("❌ No runbooks found in runbooks/ directory")
        print("   Create .md files in that directory first")
        return False
 
    print(f"\n📚 Indexing {len(runbook_files)} runbooks...")
 
    total_chunks = 0
    all_ids = []
    all_texts = []
    all_embeddings = []
    all_metadatas = []
 
    for filepath in runbook_files:
        filename = os.path.basename(filepath)
        print(f"   Processing {filename}...", end=" ")
 
        with open(filepath, "r") as f:
            content = f.read()
 
        ## Split the runbook into chunks
        chunks = chunk_text(content)
        print(f"{len(chunks)} chunks")
 
        for i, chunk in enumerate(chunks):
            chunk_id = f"{filename}_chunk_{i}"
 
            all_ids.append(chunk_id)
            all_texts.append(chunk)
            all_metadatas.append({
                "source_file": filename,
                "chunk_index": i,
                "total_chunks": len(chunks),
            })
 
        total_chunks += len(chunks)
 
    ## Encode all chunks at once (faster than one-by-one)
    print(f"\n🔢 Generating embeddings for {total_chunks} chunks...")
    all_embeddings = embedder.encode(all_texts, show_progress_bar=True).tolist()
 
    ## Add everything to ChromaDB in one batch
    collection.add(
        ids=all_ids,
        documents=all_texts,
        embeddings=all_embeddings,
        metadatas=all_metadatas,
    )
 
    print(f"\n✅ Indexed {total_chunks} chunks from {len(runbook_files)} runbooks")
    print(f"   Stored in: {CHROMA_DB_PATH}")
 
    ## Verify the index by running a test search
    print(f"\n🔍 Test search: 'pod is OOMKilled and crashing'")
    test_embedding = embedder.encode(["pod is OOMKilled and crashing"]).tolist()
    results = collection.query(query_embeddings=test_embedding, n_results=1)
    if results["documents"][0]:
        print(f"   Found: {results['metadatas'][0][0]['source_file']}")
        print(f"   Preview: {results['documents'][0][0][:100]}...")
 
    return True
 
 
if __name__ == "__main__":
    build_runbook_index()

Bash

## Index the runbooks
python3 -m src.runbook_indexer
 
## Expected output:
## 📦 Loading embedding model 'all-MiniLM-L6-v2'...
## 📚 Indexing 3 runbooks...
##    Processing oom-kill-runbook.md... 3 chunks
##    Processing high-error-rate-runbook.md... 3 chunks
##    Processing pod-crash-loop-runbook.md... 3 chunks
## 🔢 Generating embeddings for 9 chunks...
## ✅ Indexed 9 chunks from 3 runbooks
## 🔍 Test search: 'pod is OOMKilled and crashing'
##    Found: oom-kill-runbook.md
 
## Check what was created
ls -lh data/chroma/

Building the RAG Retriever

Searching runbooks by meaning

PYTHON

# src/rag_retriever.py
# Searches the ChromaDB vector index for runbook chunks
# relevant to a given alert description.
 
import chromadb
from sentence_transformers import SentenceTransformer
from src.config import (
    CHROMA_DB_PATH, CHROMA_COLLECTION_NAME,
    EMBEDDING_MODEL, RAG_TOP_K
)
 
## Load the embedder and ChromaDB client once at module load time
## This avoids reloading the 90MB model on every search
_embedder = None
_collection = None
 
 
def _get_embedder():
    """Lazy-load the embedding model (only loads once per process)."""
    global _embedder
    if _embedder is None:
        _embedder = SentenceTransformer(EMBEDDING_MODEL)
    return _embedder
 
 
def _get_collection():
    """Lazy-load the ChromaDB collection (only connects once per process)."""
    global _collection
    if _collection is None:
        client = chromadb.PersistentClient(path=CHROMA_DB_PATH)
        _collection = client.get_collection(CHROMA_COLLECTION_NAME)
    return _collection
 
 
def search_runbooks(query_text, top_k=RAG_TOP_K):
    """
    Search the runbook index for chunks most relevant to the query.
 
    Parameters:
        query_text: the alert description or diagnostic question
        top_k: how many chunks to return (default from config)
 
    Returns:
        list of dicts, each containing:
          - text: the runbook chunk text
          - source: which runbook file it came from
          - distance: similarity score (lower = more similar for cosine)
    """
    embedder = _get_embedder()
    collection = _get_collection()
 
    ## Convert the query to a vector using the same model used during indexing
    ## IMPORTANT: must use identical model -- different models produce
    ## incompatible vector spaces and search will return nonsense
    query_embedding = embedder.encode([query_text]).tolist()
 
    ## Search ChromaDB for the nearest vectors
    results = collection.query(
        query_embeddings=query_embedding,
        n_results=top_k,
        include=["documents", "metadatas", "distances"],
    )
 
    if not results["documents"][0]:
        return []
 
    ## Format results for easy consumption
    formatted = []
    for i in range(len(results["documents"][0])):
        formatted.append({
            "text": results["documents"][0][i],
            "source": results["metadatas"][0][i]["source_file"],
            "distance": results["distances"][0][i],
        })
 
    return formatted
 
 
def format_runbooks_for_prompt(results):
    """
    Convert RAG results into a clean string for the LLM prompt.
    Each chunk is labelled with its source runbook for attribution.
    """
    if not results:
        return "No relevant runbooks found."
 
    sections = []
    for i, result in enumerate(results, 1):
        sections.append(
            f"--- Runbook {i}: {result['source']} (relevance score: {1 - result['distance']:.2f}) ---\n"
            f"{result['text']}"
        )
 
    return "\n\n".join(sections)

Building the Context Collector

Getting live infrastructure state

The context collector queries Prometheus for the current state of the affected service. This is what transforms the agent from "generic advice giver" to "system-aware diagnosis engine."

PYTHON

# src/context_collector.py
# Queries Prometheus for live metrics about the affected service
# and formats them as structured context for the LLM.
 
import requests
import subprocess
from datetime import datetime
from src.config import PROMETHEUS_URL
 
 
def query_prometheus(promql_query):
    """
    Run a single PromQL query and return the scalar result.
    Returns None if the query fails or returns no data.
    """
    try:
        response = requests.get(
            f"{PROMETHEUS_URL}/api/v1/query",
            params={"query": promql_query},
            timeout=10,
        )
        data = response.json()
 
        if data["status"] != "success" or not data["data"]["result"]:
            return None
 
        ## Return the value from the first result
        return float(data["data"]["result"][0]["value"][1])
 
    except Exception:
        return None
 
 
def collect_service_context(namespace, service_name, alert_name):
    """
    Collect live metrics and events for the affected service.
 
    Parameters:
        namespace: Kubernetes namespace where the service runs
        service_name: name of the deployment/service that triggered the alert
        alert_name: name of the Alertmanager alert (used to tailor queries)
 
    Returns:
        dict containing all collected context, formatted as strings
    """
 
    context = {
        "collected_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
        "namespace": namespace,
        "service_name": service_name,
        "alert_name": alert_name,
    }
 
    ## ── CPU Metrics ──────────────────────────────────────────
    cpu_usage = query_prometheus(
        f'avg(rate(container_cpu_usage_seconds_total{{container="{service_name}",namespace="{namespace}"}}[5m]))'
    )
    cpu_limit = query_prometheus(
        f'avg(kube_pod_container_resource_limits{{container="{service_name}",namespace="{namespace}",resource="cpu"}})'
    )
 
    if cpu_usage is not None:
        context["cpu_usage_cores"] = round(cpu_usage, 4)
        if cpu_limit and cpu_limit > 0:
            context["cpu_utilization_pct"] = round((cpu_usage / cpu_limit) * 100, 1)
 
    ## ── Memory Metrics ───────────────────────────────────────
    mem_usage = query_prometheus(
        f'avg(container_memory_working_set_bytes{{container="{service_name}",namespace="{namespace}"}})'
    )
    mem_limit = query_prometheus(
        f'avg(kube_pod_container_resource_limits{{container="{service_name}",namespace="{namespace}",resource="memory"}})'
    )
 
    if mem_usage is not None:
        context["memory_usage_mb"] = round(mem_usage / (1024 * 1024), 1)
        if mem_limit and mem_limit > 0:
            context["memory_utilization_pct"] = round((mem_usage / mem_limit) * 100, 1)
            context["memory_limit_mb"] = round(mem_limit / (1024 * 1024), 1)
 
    ## ── HTTP Metrics ─────────────────────────────────────────
    http_rps = query_prometheus(
        f'sum(rate(http_requests_total{{service="{service_name}"}}[5m]))'
    )
    error_rate = query_prometheus(
        f'sum(rate(http_requests_total{{service="{service_name}",status=~"5.."}}[5m])) / '
        f'sum(rate(http_requests_total{{service="{service_name}"}}[5m]))'
    )
    p95_latency = query_prometheus(
        f'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{{service="{service_name}"}}[5m])) by (le))'
    )
 
    if http_rps is not None:
        context["http_requests_per_second"] = round(http_rps, 2)
    if error_rate is not None:
        context["http_error_rate_pct"] = round(error_rate * 100, 2)
    if p95_latency is not None:
        context["http_p95_latency_ms"] = round(p95_latency * 1000, 1)
 
    ## ── Pod Status via kubectl ───────────────────────────────
    ## kubectl gives us restart counts and current pod states
    ## which Prometheus does not easily provide in structured form
    try:
        result = subprocess.run(
            ["kubectl", "get", "pods", "-n", namespace,
             "-l", f"app={service_name}",
             "--no-headers",
             "-o", "custom-columns=NAME:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[0].restartCount"],
            capture_output=True, text=True, timeout=10
        )
        if result.returncode == 0 and result.stdout.strip():
            context["pod_status"] = result.stdout.strip()
    except Exception:
        context["pod_status"] = "kubectl unavailable"
 
    ## ── Recent Events ────────────────────────────────────────
    ## Events show OOMKills, failed image pulls, probe failures
    try:
        result = subprocess.run(
            ["kubectl", "get", "events", "-n", namespace,
             "--field-selector", f"involvedObject.name={service_name}",
             "--sort-by=.metadata.creationTimestamp",
             "-o", "custom-columns=TIME:.metadata.creationTimestamp,REASON:.reason,MESSAGE:.message"],
            capture_output=True, text=True, timeout=10
        )
        if result.returncode == 0 and result.stdout.strip():
            ## Take only the last 5 events to keep the context concise
            lines = result.stdout.strip().split("\n")
            context["recent_events"] = "\n".join(lines[-5:])
    except Exception:
        context["recent_events"] = "kubectl unavailable"
 
    return context
 
 
def format_context_for_prompt(context):
    """
    Format the collected context as a structured string for the LLM prompt.
    The LLM reads this to understand what is happening right now.
    """
 
    lines = [
        f"Service: {context['service_name']} in namespace {context['namespace']}",
        f"Alert: {context['alert_name']}",
        f"Data collected at: {context['collected_at']}",
        "",
        "=== Current Infrastructure State ===",
    ]
 
    if "cpu_usage_cores" in context:
        cpu_line = f"CPU Usage: {context['cpu_usage_cores']} cores"
        if "cpu_utilization_pct" in context:
            cpu_line += f" ({context['cpu_utilization_pct']}% of limit)"
        lines.append(cpu_line)
 
    if "memory_usage_mb" in context:
        mem_line = f"Memory Usage: {context['memory_usage_mb']} MB"
        if "memory_limit_mb" in context:
            mem_line += f" / {context['memory_limit_mb']} MB limit ({context.get('memory_utilization_pct', '?')}%)"
        lines.append(mem_line)
 
    if "http_requests_per_second" in context:
        lines.append(f"HTTP RPS: {context['http_requests_per_second']}")
    if "http_error_rate_pct" in context:
        lines.append(f"HTTP Error Rate: {context['http_error_rate_pct']}%")
    if "http_p95_latency_ms" in context:
        lines.append(f"P95 Latency: {context['http_p95_latency_ms']} ms")
 
    if "pod_status" in context:
        lines.extend(["", "=== Pod Status ===", context["pod_status"]])
 
    if "recent_events" in context:
        lines.extend(["", "=== Recent Kubernetes Events ===", context["recent_events"]])
 
    return "\n".join(lines)

Building the Agent Core

The reasoning loop

This is where everything comes together. The agent receives an alert, collects context, retrieves runbooks, and asks the LLM to synthesise a diagnosis.

PYTHON

# src/agent.py
# The main reasoning loop.
# Receives parsed alert data, collects context, runs RAG,
# calls the LLM, and returns a structured response.
 
import requests
import json
import logging
import os
from src.config import OLLAMA_URL, LLM_MODEL, LOG_PATH
from src.context_collector import collect_service_context, format_context_for_prompt
from src.rag_retriever import search_runbooks, format_runbooks_for_prompt
 
os.makedirs(os.path.dirname(LOG_PATH), exist_ok=True)
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[
        logging.FileHandler(LOG_PATH),
        logging.StreamHandler(),
    ],
)
logger = logging.getLogger(__name__)
 
 
def call_llm(prompt, max_tokens=1000):
    """
    Call the local Ollama LLM with a prompt and return the response text.
 
    Parameters:
        prompt: the full text prompt to send
        max_tokens: maximum response length (keep short for speed)
 
    Returns:
        response string, or None if the call failed
    """
    try:
        response = requests.post(
            f"{OLLAMA_URL}/api/generate",
            json={
                "model": LLM_MODEL,
                "prompt": prompt,
                "stream": False,   ## wait for full response before returning
                "options": {
                    "num_predict": max_tokens,
                    "temperature": 0.1,   ## low temperature = more deterministic, better for ops
                }
            },
            timeout=120,   ## LLM can take up to 2 minutes on slow hardware
        )
        data = response.json()
        return data.get("response", "").strip()
 
    except requests.exceptions.ConnectionError:
        logger.error(f"Cannot reach Ollama at {OLLAMA_URL}")
        logger.error("Start Ollama with: ollama serve")
        return None
    except Exception as e:
        logger.error(f"LLM call failed: {e}")
        return None
 
 
def build_diagnosis_prompt(alert_data, context_text, runbook_text):
    """
    Build the LLM prompt that combines alert data, live context, and runbooks.
 
    The prompt structure matters here:
    1. Role: tell the LLM what it is
    2. Context: give it the live infrastructure state
    3. Runbooks: give it the relevant remediation knowledge
    4. Task: tell it exactly what to produce
    5. Format: specify the output format strictly
 
    This structured approach produces consistent, parseable output.
    """
 
    prompt = f"""You are an experienced Site Reliability Engineer at a fast-growing Indian fintech company.
You are responding to a production alert at {alert_data.get('starts_at', 'unknown time')}.
 
Your job is to diagnose this incident and provide clear remediation steps.
Be specific. Be actionable. Reference the actual metric values in your diagnosis.
 
=== ALERT DETAILS ===
Alert Name: {alert_data.get('alert_name', 'unknown')}
Severity: {alert_data.get('severity', 'unknown')}
Summary: {alert_data.get('summary', 'No summary provided')}
Description: {alert_data.get('description', 'No description provided')}
 
=== LIVE INFRASTRUCTURE STATE ===
{context_text}
 
=== RELEVANT RUNBOOKS FROM KNOWLEDGE BASE ===
{runbook_text}
 
=== YOUR TASK ===
Based on the alert details, the live infrastructure state, and the runbooks above, provide:
 
1. DIAGNOSIS (2-3 sentences): What is most likely happening and why, based on the specific metric values you see.
 
2. IMMEDIATE ACTIONS (numbered list): The exact kubectl or remediation commands to run right now, in the correct order. Be specific -- include the actual namespace and service name from the context.
 
3. ESCALATE IF (1-2 sentences): The specific condition under which this should be escalated to the engineering lead.
 
4. ROOT CAUSE HYPOTHESIS (1 sentence): Your best guess at the underlying root cause.
 
Keep your response structured with these exact four headers. Be concise."""
 
    return prompt
 
 
def run_agent(alert_payload):
    """
    Main agent entry point.
    Takes a parsed Alertmanager webhook payload and returns a diagnosis dict.
 
    Parameters:
        alert_payload: dict from the Alertmanager webhook
 
    Returns:
        dict with keys: diagnosis, immediate_actions, escalate_if, root_cause, raw_response
    """
 
    alert_name = alert_payload.get("alert_name", "UnknownAlert")
    namespace = alert_payload.get("namespace", "default")
    service_name = alert_payload.get("service", "unknown-service")
 
    logger.info(f"🚨 Processing alert: {alert_name} for {service_name} in {namespace}")
 
    ## ── Step 1: Collect live context ─────────────────────────
    logger.info("   Collecting infrastructure context from Prometheus...")
    context = collect_service_context(namespace, service_name, alert_name)
    context_text = format_context_for_prompt(context)
    logger.info(f"   Context collected: {len(context)} data points")
 
    ## ── Step 2: RAG search for relevant runbooks ─────────────
    ## Build a search query from the alert name and description
    search_query = f"{alert_name} {alert_payload.get('description', '')} {alert_payload.get('summary', '')}"
    logger.info(f"   Searching runbooks for: '{search_query[:60]}...'")
 
    runbook_results = search_runbooks(search_query)
    runbook_text = format_runbooks_for_prompt(runbook_results)
 
    if runbook_results:
        logger.info(f"   Found {len(runbook_results)} relevant runbook sections:")
        for r in runbook_results:
            logger.info(f"     - {r['source']} (score: {1 - r['distance']:.2f})")
    else:
        logger.warning("   No relevant runbooks found")
 
    ## ── Step 3: Build and send the LLM prompt ─────────────────
    prompt = build_diagnosis_prompt(alert_payload, context_text, runbook_text)
    logger.info(f"   Calling LLM ({LLM_MODEL})...")
 
    llm_response = call_llm(prompt)
 
    if not llm_response:
        return {
            "error": "LLM call failed",
            "context": context_text,
            "runbooks_found": [r["source"] for r in runbook_results],
        }
 
    logger.info(f"✅ Diagnosis complete ({len(llm_response)} chars)")
 
    return {
        "alert_name": alert_name,
        "service": service_name,
        "namespace": namespace,
        "context_summary": context_text,
        "runbooks_used": [r["source"] for r in runbook_results],
        "diagnosis": llm_response,
        "raw_alert": alert_payload,
    }

Building the Webhook Receiver

Receiving alerts from Alertmanager

PYTHON

# src/webhook_receiver.py
# FastAPI server that receives Alertmanager webhook payloads
# and triggers the agent for each alert.
 
import logging
from fastapi import FastAPI, Request, BackgroundTasks
from src.agent import run_agent
from src.notifier import send_to_slack
from src.config import WEBHOOK_HOST, WEBHOOK_PORT
 
logger = logging.getLogger(__name__)
app = FastAPI(title="AIOps Incident Response Agent")
 
 
def parse_alertmanager_payload(payload):
    """
    Extract useful fields from the Alertmanager webhook JSON.
 
    Alertmanager sends one webhook with potentially multiple alerts.
    This function returns a list, one item per alert.
    """
    alerts = []
 
    for alert in payload.get("alerts", []):
        labels = alert.get("labels", {})
        annotations = alert.get("annotations", {})
 
        parsed = {
            "alert_name": labels.get("alertname", "UnknownAlert"),
            "severity": labels.get("severity", "warning"),
            "namespace": labels.get("namespace", "default"),
            "service": labels.get("service", labels.get("app", "unknown")),
            "summary": annotations.get("summary", ""),
            "description": annotations.get("description", ""),
            "starts_at": alert.get("startsAt", ""),
            "status": alert.get("status", "firing"),
            "labels": labels,
        }
        alerts.append(parsed)
 
    return alerts
 
 
@app.post("/webhook")
async def receive_alert(request: Request, background_tasks: BackgroundTasks):
    """
    Receive an Alertmanager webhook and trigger the agent asynchronously.
 
    Running the agent in the background means Alertmanager gets an
    immediate 200 OK response and does not time out while waiting
    for the LLM (which can take 30-60 seconds).
    """
    payload = await request.json()
    alerts = parse_alertmanager_payload(payload)
 
    logger.info(f"📨 Received webhook with {len(alerts)} alert(s)")
 
    for alert in alerts:
        if alert["status"] == "resolved":
            ## Send a simple resolved notification without running the agent
            background_tasks.add_task(
                send_to_slack,
                f"✅ *Resolved:* {alert['alert_name']} for `{alert['service']}` in `{alert['namespace']}`",
                severity="good",
            )
        else:
            ## Run the full agent pipeline in the background
            background_tasks.add_task(process_alert, alert)
 
    return {"status": "received", "alert_count": len(alerts)}
 
 
async def process_alert(alert_data):
    """Run the full agent pipeline and send the result to Slack."""
    result = run_agent(alert_data)
 
    if "error" in result:
        await send_to_slack(
            f"⚠️ Agent error for {alert_data['alert_name']}: {result['error']}",
            severity="warning",
        )
        return
 
    ## Format a structured Slack message
    slack_message = format_slack_message(result)
    await send_to_slack(slack_message, severity="alert")
 
 
def format_slack_message(result):
    """Format the agent result as a structured Slack message."""
    runbooks = ", ".join(result.get("runbooks_used", []))
    if not runbooks:
        runbooks = "None found"
 
    return f"""🚨 *Incident Response: {result['alert_name']}*
Service: `{result['service']}` | Namespace: `{result['namespace']}`
Runbooks consulted: {runbooks}
 
{result['diagnosis']}
 
_Generated by AIOps Incident Response Agent_"""
 
 
@app.get("/health")
async def health():
    return {"status": "ok"}

Building the Slack Notifier

PYTHON

# src/notifier.py
# Sends formatted messages to a Slack channel via incoming webhook.
 
import requests
import logging
from src.config import SLACK_WEBHOOK_URL
 
logger = logging.getLogger(__name__)
 
 
async def send_to_slack(message, severity="default"):
    """
    Send a message to the Slack channel configured in SLACK_WEBHOOK_URL.
 
    Parameters:
        message: the text to send (supports Slack markdown)
        severity: "alert" (red), "warning" (yellow), "good" (green), "default"
    """
 
    if not SLACK_WEBHOOK_URL:
        ## If no Slack webhook configured, just log the message
        ## This allows the agent to work without Slack during development
        logger.info(f"📢 [SLACK NOT CONFIGURED] {message}")
        return True
 
    color_map = {
        "alert": "#E53E3E",    ## red
        "warning": "#F6AD55",  ## orange
        "good": "#48BB78",     ## green
        "default": "#4A5568",  ## gray
    }
 
    payload = {
        "attachments": [
            {
                "color": color_map.get(severity, color_map["default"]),
                "text": message,
                "mrkdwn_in": ["text"],
            }
        ]
    }
 
    try:
        response = requests.post(
            SLACK_WEBHOOK_URL,
            json=payload,
            timeout=10,
        )
        if response.status_code == 200:
            logger.info("   ✅ Slack message sent")
            return True
        else:
            logger.warning(f"   ⚠️ Slack returned {response.status_code}: {response.text}")
            return False
 
    except Exception as e:
        logger.error(f"   ❌ Slack send failed: {e}")
        return False

Running and Testing the Agent

Start all the services

Bash

## Terminal 1: Start Ollama (if not already running)
ollama serve
 
## Terminal 2: Port-forward Prometheus and Alertmanager
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 &
kubectl port-forward -n monitoring svc/alertmanager-operated 9093:9093 &
 
## Terminal 3: Index the runbooks
python3 -m src.runbook_indexer
 
## Terminal 4: Start the webhook receiver
python3 -c "
import uvicorn
from src.webhook_receiver import app
from src.config import WEBHOOK_HOST, WEBHOOK_PORT
uvicorn.run(app, host=WEBHOOK_HOST, port=WEBHOOK_PORT)
"
 
## Expected output:
## INFO:     Started server process
## INFO:     Waiting for application startup.
## INFO:     Application startup complete.
## INFO:     Uvicorn running on http://0.0.0.0:8080

Test with a simulated alert

Bash

## Send a test alert directly to the webhook endpoint
## This simulates what Alertmanager would send
curl -X POST http://localhost:8080/webhook \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4",
    "groupKey": "test-group",
    "status": "firing",
    "alerts": [
      {
        "status": "firing",
        "labels": {
          "alertname": "PodOOMKilled",
          "severity": "warning",
          "namespace": "production",
          "service": "payment-service",
          "app": "payment-service"
        },
        "annotations": {
          "summary": "Pod payment-service was OOMKilled",
          "description": "Container payment-service in namespace production was killed due to memory limit. Memory usage was at 498Mi / 512Mi limit."
        },
        "startsAt": "2024-01-15T03:23:00Z"
      }
    ]
  }'
 
## Expected immediate response:
## {"status": "received", "alert_count": 1}
 
## Watch Terminal 4 for the agent processing:
## 🚨 Processing alert: PodOOMKilled for payment-service in production
##    Collecting infrastructure context from Prometheus...
##    Searching runbooks for: 'PodOOMKilled Pod payment-service...'
##    Found 2 relevant runbook sections:
##      - oom-kill-runbook.md (score: 0.89)
##    Calling LLM (llama3.1)...
## ✅ Diagnosis complete (847 chars)

Configure Alertmanager to send real alerts

YAML

## Add this to your Alertmanager config (alertmanager.yaml)
## to send all firing alerts to the agent
 
route:
  receiver: aiops-agent
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
 
receivers:
  - name: aiops-agent
    webhook_configs:
      - url: http://aiops-incident-agent.monitoring.svc:8080/webhook
        send_resolved: true

Bash

## Apply the updated Alertmanager config
kubectl create configmap alertmanager-config \
  --from-file=alertmanager.yaml \
  -n monitoring \
  --dry-run=client -o yaml | kubectl apply -f -
 
## Reload Alertmanager
kubectl rollout restart deployment/alertmanager -n monitoring

Deploy to Kubernetes

YAML

## k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: aiops-incident-agent
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aiops-incident-agent
  template:
    metadata:
      labels:
        app: aiops-incident-agent
    spec:
      containers:
      - name: agent
        image: your-registry/aiops-incident-agent:v1.0.0
        ports:
        - containerPort: 8080
        env:
        - name: PROMETHEUS_URL
          value: "http://prometheus-operated.monitoring.svc:9090"
        - name: OLLAMA_URL
          ## Ollama runs as a separate deployment in the cluster
          value: "http://ollama.monitoring.svc:11434"
        - name: SLACK_WEBHOOK_URL
          valueFrom:
            secretKeyRef:
              name: aiops-secrets
              key: slack-webhook-url
        resources:
          requests:
            cpu: "200m"
            memory: "512Mi"
          limits:
            cpu: "1000m"
            memory: "1Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: aiops-incident-agent
  namespace: monitoring
spec:
  selector:
    app: aiops-incident-agent
  ports:
  - port: 8080
    targetPort: 8080

Bash

## Create the Slack webhook secret
kubectl create secret generic aiops-secrets \
  --from-literal=slack-webhook-url=https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -n monitoring
 
## Build and deploy
docker build -t your-registry/aiops-incident-agent:v1.0.0 .
docker push your-registry/aiops-incident-agent:v1.0.0
kubectl apply -f k8s/deployment.yaml
 
## Verify
kubectl get pods -n monitoring | grep aiops-incident-agent
kubectl logs -n monitoring deployment/aiops-incident-agent -f

Production Checklist

Bash

## ─── 1. Runbook index is populated ──────────────────────────
python3 -c "
import chromadb
client = chromadb.PersistentClient(path='data/chroma')
col = client.get_collection('runbooks')
print(f'Runbook chunks indexed: {col.count()}')
"
## Should show a number > 0
 
## ─── 2. RAG search returns relevant results ──────────────────
python3 -c "
from src.rag_retriever import search_runbooks
results = search_runbooks('pod OOMKilled memory limit exceeded')
for r in results:
    print(f'{r[\"source\"]} (score: {1-r[\"distance\"]:.2f})')
    print(f'  {r[\"text\"][:80]}...')
"
## Should return oom-kill-runbook.md with a high score
 
## ─── 3. Ollama responds ──────────────────────────────────────
curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3.1","prompt":"Say OK","stream":false}' | python3 -m json.tool | grep response
## Should return a short response
 
## ─── 4. Webhook receiver is healthy ─────────────────────────
curl http://localhost:8080/health
## Should return {"status": "ok"}
 
## ─── 5. Full pipeline test ───────────────────────────────────
## Send the test alert from the Testing section above
## Watch logs for: "✅ Diagnosis complete"
 
echo "✅ Production checklist complete"

Common Production Mistakes

Loading the embedding model on every request. The SentenceTransformer model is 90MB. If you initialise it inside the search function instead of at module load time, every single alert triggers a 3-5 second model load before anything happens. The fix is to initialise _embedder = None at module level and load it lazily the first time it is needed, as shown in the code above. After the first request warms it up, all subsequent searches take milliseconds.

Using a different embedding model for indexing and searching. If you index runbooks with all-MiniLM-L6-v2 and then accidentally search with all-mpnet-base-v2, the vectors are in completely different spaces. The similarity scores will be meaningless — you will get random runbooks regardless of the query. Always use the same model in runbook_indexer.py and rag_retriever.py. The EMBEDDING_MODEL config variable exists specifically to ensure this.

Setting temperature too high on the LLM. At temperature=0.8, the LLM is creative. That is great for writing. For incident response, creativity means making up remediation steps. Setting temperature=0.1 forces the model to stay grounded in the runbook content it was given. If your agent is suggesting bizarre commands that do not match your runbooks, check the temperature setting first.

Not indexing runbooks before starting the agent. If ChromaDB has no data, search_runbooks returns nothing. The LLM then has only the live metrics to reason from and will give generic advice. Always run python3 -m src.runbook_indexer before starting the webhook server. In production, add an init container to the Kubernetes deployment that indexes runbooks before the main container starts.

Running multiple replicas of the agent. With replicas: 1 the agent receives each alert once. With replicas: 2, both pods receive the webhook and the LLM runs twice, sending two Slack messages for every alert. The webhook receiver does not have distributed deduplication. Keep replicas at 1. Use a liveness probe to restart it if it becomes unhealthy rather than running two copies.

Not adding new runbooks after incidents. The agent is only as good as its runbook library. After every real incident, write a runbook for it and re-run the indexer. Teams that treat the runbook library as a living document get better and better responses over time. Teams that index once and never add runbooks find the agent increasingly unable to handle new failure modes.

Quick Reference

Component	File	Runs When
`runbook_indexer.py`	`src/runbook_indexer.py`	Once at setup, re-run when runbooks change
`webhook_receiver.py`	`src/webhook_receiver.py`	Always running, receives alerts
`context_collector.py`	`src/context_collector.py`	On every alert
`rag_retriever.py`	`src/rag_retriever.py`	On every alert
`agent.py`	`src/agent.py`	On every alert
`notifier.py`	`src/notifier.py`	On every alert after diagnosis

Command	What It Does
`python3 -m src.runbook_indexer`	Index all runbooks into ChromaDB
`curl http://localhost:8080/health`	Check if webhook server is running
`curl -X POST localhost:8080/webhook -d '{...}'`	Send a test alert
`ollama list`	Check which LLM models are available
`ollama serve`	Start the Ollama API server
`kubectl logs deployment/aiops-incident-agent -f`	Watch agent logs in production

Videos & Guides

No external walkthroughs, video tutorials, or reference files attached to this blueprint yet.