### What you are building and why it matters Swiggy processes roughly 1.5 million orders a day. During peak dinner hours, every service matters. When the `order-service` pods start getting OOMKilled, engineers have about three minutes before users see timeouts, before orders fail silently, before the support queue starts filling up. In those three minutes, a human on-call engineer needs to: 1. Get paged and wake up (or stop what they are doing) 2. Open the monitoring dashboard 3. Figure out what is wrong 4. Look up the runbook 5. Run the remediation commands 6. Verify the fix worked Most of that time is not thinking — it is plumbing. Finding the right Prometheus graph. Navigating to the right runbook. Running commands they have run dozens of times before. Self-healing infrastructure eliminates that plumbing entirely. The system detects the anomaly, diagnoses the cause, selects the remediation, executes it, and verifies the result — all before the on-call engineer's phone has even finished ringing. This capstone builds that system by wiring together everything from this roadmap: * **Capstone 1's Isolation Forest model** detects the anomaly from Prometheus metrics * **Capstone 2's RAG + LLM pipeline** diagnoses the cause and selects the remediation * **Ansible playbooks** execute the fix safely with full audit logging * **A decision engine** decides when to act autonomously vs. when to ask for human approval The result is a system that handles the 80% of incidents that are predictable and repetitive — restarts, scaling events, cache flushes, pod evictions — automatically. Engineers handle the remaining 20% that genuinely require judgement. Time to complete: 5-6 hours. **What you need before starting:** * Completed Capstone 1 (trained Isolation Forest model at `models/isolation_forest.joblib`) * Completed Capstone 2 (runbooks indexed in ChromaDB, Ollama running) * Ansible installed locally ```bash ## Verify Capstone 1 model exists ls -lh models/isolation_forest.joblib ## Should show the file ## Verify Capstone 2 runbooks are indexed python3 -c " import chromadb c = chromadb.PersistentClient(path='data/chroma') col = c.get_collection('runbooks') print(f'Runbooks indexed: {col.count()} chunks') " ## Verify Ollama is running curl -s http://localhost:11434/api/tags | python3 -m json.tool | grep name ## Should show at least one model ## Install Ansible pip3 install ansible --break-system-packages ansible --version echo "✅ Ready to build Capstone 3" ```
### How all three capstones connect This is the architecture you are building. Read it carefully before touching any code. ┌─────────────────────────────────────────────────────────────┐ │ DETECTION (from Capstone 1) │ │ │ │ Prometheus metrics → Isolation Forest → Anomaly score │ │ Runs every 60 seconds │ └──────────────────────────────┬──────────────────────────────┘ │ anomaly detected ▼ ┌─────────────────────────────────────────────────────────────┐ │ DIAGNOSIS (from Capstone 2) │ │ │ │ Live context (Prometheus + kubectl) → RAG runbook search │ │ → LLM reasoning → structured diagnosis with action type │ └──────────────────────────────┬──────────────────────────────┘ │ diagnosis + action ▼ ┌─────────────────────────────────────────────────────────────┐ │ DECISION ENGINE (new in Capstone 3) │ │ │ │ Is this action on the approved list? │ │ Is confidence above the threshold? │ │ Has this action run too many times recently? │ │ ├── YES → execute autonomously │ │ └── NO → send to Slack for human approval │ └──────────────────────────────┬──────────────────────────────┘ │ approved ▼ ┌─────────────────────────────────────────────────────────────┐ │ REMEDIATION (new in Capstone 3) │ │ │ │ Ansible playbook executes the fix │ │ Full audit log written │ │ Verification check runs after completion │ │ Result posted to Slack │ └─────────────────────────────────────────────────────────────┘ The **Decision Engine** is the most important new piece. Without it, you have a system that might restart pods, scale deployments, or flush caches based on a wrong diagnosis. The decision engine defines what the system is allowed to do on its own — and forces human approval for anything riskier. ### The three tiers of autonomy Not every remediation action carries the same risk. The system organises actions into three tiers: **Tier 1 — Fully autonomous (system acts immediately):** Pod restart, cache flush, horizontal scale-up. These are reversible, low-risk, and run hundreds of times a year. Automating them saves the most on-call time with the least risk. **Tier 2 — Autonomous with notification (system acts, then tells humans):** Memory limit increase, rolling restart of a deployment, draining a node. Higher impact but still reversible. The system acts immediately and posts the result to Slack so engineers can review. **Tier 3 — Human approval required (system asks before acting):** Rollback of a recent deployment, scaling down a cluster, any action involving databases or persistent storage. These actions can cause data loss or downtime if wrong. A human must approve via Slack button before the system proceeds.
### Setting up from the capstone 1 and 2 directories ```bash ## This capstone lives in a new directory but imports from capstones 1 and 2 mkdir aiops-self-healing && cd aiops-self-healing mkdir -p src ansible/playbooks ansible/inventory logs audit k8s touch src/config.py touch src/orchestrator.py touch src/decision_engine.py touch src/ansible_executor.py touch src/verifier.py touch src/audit_logger.py touch src/slack_approvals.py touch requirements.txt echo "✅ Project structure created" ``` aiops-self-healing/ src/ config.py <- all settings including autonomy thresholds orchestrator.py <- the main loop: detect -> diagnose -> decide -> act decision_engine.py <- decides autonomous vs human-approval actions ansible_executor.py <- runs Ansible playbooks safely verifier.py <- confirms the fix worked after execution audit_logger.py <- writes every action to a tamper-evident log slack_approvals.py <- sends approval requests and handles responses ansible/ playbooks/ <- one playbook per remediation action inventory/ <- Kubernetes inventory for Ansible logs/ <- orchestrator logs audit/ <- audit trail (JSONL, one entry per action) k8s/ <- Kubernetes deployment manifests ```text # requirements.txt # Inherits all deps from capstones 1 and 2 pandas==2.1.0 scikit-learn==1.3.0 requests==2.31.0 chromadb==0.4.18 sentence-transformers==2.2.2 joblib==1.3.2 fastapi==0.104.1 uvicorn==0.24.0 ansible-runner==2.3.4 schedule==1.2.0 ``` ```bash pip3 install -r requirements.txt echo "✅ Dependencies installed" ```
### The config file controls what the system is allowed to do ```python # src/config.py import os ## ── Inherited from Capstone 1 ──────────────────────────────── PROMETHEUS_URL = os.getenv("PROMETHEUS_URL", "http://localhost:9090") MODEL_PATH = "../aiops-anomaly-detection/models/isolation_forest.joblib" DATA_PATH = "../aiops-anomaly-detection/data/metrics.csv" ## ── Inherited from Capstone 2 ──────────────────────────────── CHROMA_DB_PATH = "../aiops-incident-agent/data/chroma" CHROMA_COLLECTION_NAME = "runbooks" EMBEDDING_MODEL = "all-MiniLM-L6-v2" OLLAMA_URL = os.getenv("OLLAMA_URL", "http://localhost:11434") LLM_MODEL = os.getenv("LLM_MODEL", "llama3.1") ## ── Detection Settings ─────────────────────────────────────── ## How often to run the detection loop (seconds) DETECTION_INTERVAL_SECONDS = 60 ## How many consecutive anomaly readings before triggering remediation ANOMALY_CONSECUTIVE_THRESHOLD = 2 ## ── Slack ───────────────────────────────────────────────────── SLACK_WEBHOOK_URL = os.getenv("SLACK_WEBHOOK_URL", "") ## For interactive approvals, you need a Slack bot token SLACK_BOT_TOKEN = os.getenv("SLACK_BOT_TOKEN", "") SLACK_APPROVAL_CHANNEL = os.getenv("SLACK_APPROVAL_CHANNEL", "#on-call") ## ── Autonomy Configuration ─────────────────────────────────── ## This is the most important section. ## It defines exactly what the system can do autonomously. ## Tier 1: System executes immediately, no human needed AUTONOMOUS_ACTIONS = [ "restart_pod", "flush_redis_cache", "scale_up_deployment", ## only scale UP, never scale down autonomously ] ## Tier 2: System executes, then notifies (no prior approval needed) NOTIFY_AFTER_ACTIONS = [ "increase_memory_limit", "rolling_restart_deployment", "evict_overloaded_node", ] ## Tier 3: System asks for approval before executing APPROVAL_REQUIRED_ACTIONS = [ "rollback_deployment", "scale_down_deployment", "delete_pvc", "modify_database_config", ] ## Minimum LLM confidence score (0-1) to act autonomously ## Below this threshold, always ask for human approval even for Tier 1 actions AUTONOMOUS_CONFIDENCE_THRESHOLD = 0.75 ## Maximum times the same action can run on the same service per hour ## Prevents the system from restart-looping a broken pod forever MAX_ACTIONS_PER_SERVICE_PER_HOUR = 3 ## ── Ansible ────────────────────────────────────────────────── ANSIBLE_PLAYBOOK_DIR = "ansible/playbooks" ANSIBLE_INVENTORY = "ansible/inventory/hosts.yaml" ## ── Audit ───────────────────────────────────────────────────── AUDIT_LOG_PATH = "audit/actions.jsonl" ## ── Verification ───────────────────────────────────────────── ## Seconds to wait after remediation before verifying the fix worked VERIFICATION_WAIT_SECONDS = 90 ``` > 📌 **Remember:** The `AUTONOMOUS_ACTIONS` list is your blast radius control. Start it small — just `restart_pod` — and expand it as you build confidence in the system's accuracy. Never add irreversible actions to this list.
### Why Ansible instead of running kubectl directly You could call `subprocess.run(["kubectl", "rollout", "restart", ...])` directly from Python. Many systems do. The problem is that direct subprocess calls have no idempotency guarantees, no retry logic, no structured output, and no native audit trail. Ansible gives you all four. Each playbook is idempotent (safe to run twice), has built-in retry with backoff, produces structured JSON output you can parse, and Ansible Runner writes every execution to a persistent artifact directory. Think of it this way: kubectl is the screwdriver. Ansible is the procedure that says "use the screwdriver, turn it this many times, verify the screw is seated, mark it done." ```bash ## Create the pod restart playbook cat > ansible/playbooks/restart_pod.yaml << 'EOF' --- - name: Restart a Kubernetes pod by deleting it (Deployment recreates it) hosts: localhost connection: local gather_facts: false vars: ## These are passed in by the Python executor: ## namespace: the Kubernetes namespace ## pod_name: the specific pod to restart ## service_name: used for verification after restart tasks: - name: Verify the pod exists before trying to delete it kubernetes.core.k8s_info: kind: Pod namespace: "{{ namespace }}" name: "{{ pod_name }}" register: pod_info failed_when: pod_info.resources | length == 0 - name: Delete the pod (the Deployment controller recreates it immediately) kubernetes.core.k8s: state: absent kind: Pod namespace: "{{ namespace }}" name: "{{ pod_name }}" - name: Wait for a new pod to reach Running state kubernetes.core.k8s_info: kind: Pod namespace: "{{ namespace }}" label_selectors: - "app={{ service_name }}" field_selectors: - "status.phase=Running" register: new_pods retries: 10 delay: 10 until: new_pods.resources | length > 0 - name: Report result debug: msg: "Pod {{ pod_name }} restarted. New pods running: {{ new_pods.resources | length }}" EOF echo "✅ restart_pod.yaml created" ``` ```bash ## Create the scale-up playbook cat > ansible/playbooks/scale_up_deployment.yaml << 'EOF' --- - name: Scale up a Kubernetes deployment hosts: localhost connection: local gather_facts: false vars: ## namespace: the Kubernetes namespace ## deployment_name: name of the Deployment to scale ## current_replicas: current replica count (passed in by the executor) ## scale_factor: multiply current replicas by this (default 1.5) tasks: - name: Get current replica count kubernetes.core.k8s_info: kind: Deployment namespace: "{{ namespace }}" name: "{{ deployment_name }}" register: deployment_info failed_when: deployment_info.resources | length == 0 - name: Calculate new replica count (current * scale_factor, minimum 2) set_fact: new_replicas: >- {{ [ (deployment_info.resources[0].spec.replicas | float * (scale_factor | default(1.5))) | round | int, 2 ] | max }} - name: Scale the deployment up kubernetes.core.k8s_scale: kind: Deployment namespace: "{{ namespace }}" name: "{{ deployment_name }}" replicas: "{{ new_replicas }}" - name: Wait for rollout to complete command: > kubectl rollout status deployment/{{ deployment_name }} -n {{ namespace }} --timeout=120s register: rollout_result - name: Report result debug: msg: "Scaled {{ deployment_name }} to {{ new_replicas }} replicas. {{ rollout_result.stdout }}" EOF echo "✅ scale_up_deployment.yaml created" ``` ```bash ## Create the memory limit increase playbook cat > ansible/playbooks/increase_memory_limit.yaml << 'EOF' --- - name: Increase memory limit for a deployment hosts: localhost connection: local gather_facts: false vars: ## namespace: Kubernetes namespace ## deployment_name: name of the Deployment ## container_name: which container to update (usually same as deployment_name) ## new_memory_limit: e.g. "512Mi" or "1Gi" tasks: - name: Verify deployment exists kubernetes.core.k8s_info: kind: Deployment namespace: "{{ namespace }}" name: "{{ deployment_name }}" register: dep_info failed_when: dep_info.resources | length == 0 - name: Patch the memory limit kubernetes.core.k8s_json_patch: kind: Deployment namespace: "{{ namespace }}" name: "{{ deployment_name }}" patch: - op: replace path: "/spec/template/spec/containers/0/resources/limits/memory" value: "{{ new_memory_limit }}" - name: Trigger rolling restart for the new limit to take effect command: > kubectl rollout restart deployment/{{ deployment_name }} -n {{ namespace }} - name: Wait for rollout command: > kubectl rollout status deployment/{{ deployment_name }} -n {{ namespace }} --timeout=180s - name: Report result debug: msg: "Memory limit for {{ deployment_name }} updated to {{ new_memory_limit }}" EOF echo "✅ increase_memory_limit.yaml created" ``` ```bash ## Create the rollback playbook (Tier 3 — requires approval) cat > ansible/playbooks/rollback_deployment.yaml << 'EOF' --- - name: Roll back a Kubernetes deployment to the previous revision hosts: localhost connection: local gather_facts: false vars: ## namespace: Kubernetes namespace ## deployment_name: which deployment to roll back tasks: - name: Show rollout history before rolling back command: > kubectl rollout history deployment/{{ deployment_name }} -n {{ namespace }} register: history - name: Log the history for audit purposes debug: msg: "{{ history.stdout }}" - name: Execute rollback to previous revision command: > kubectl rollout undo deployment/{{ deployment_name }} -n {{ namespace }} register: rollback_result - name: Wait for rollback to complete command: > kubectl rollout status deployment/{{ deployment_name }} -n {{ namespace }} --timeout=180s - name: Report result debug: msg: "Rollback complete. {{ rollback_result.stdout }}" EOF echo "✅ rollback_deployment.yaml created" ``` ```bash ## Install the Kubernetes Ansible collection ansible-galaxy collection install kubernetes.core ## Create the inventory file cat > ansible/inventory/hosts.yaml << 'EOF' all: hosts: localhost: ansible_connection: local ansible_python_interpreter: "{{ ansible_playbook_python }}" EOF echo "✅ Ansible setup complete" ```
### Running playbooks safely from Python ```python # src/ansible_executor.py # Executes Ansible playbooks and returns structured results. # Uses ansible-runner which gives us async execution, structured # output, and persistent artifacts for auditing. import ansible_runner import os import json import logging from src.config import ANSIBLE_PLAYBOOK_DIR, ANSIBLE_INVENTORY logger = logging.getLogger(__name__) def run_playbook(playbook_name, extra_vars, private_data_dir="ansible/artifacts"): """ Execute an Ansible playbook with the given variables. Parameters: playbook_name: filename of the playbook (e.g. "restart_pod.yaml") extra_vars: dict of variables to pass to the playbook private_data_dir: where ansible-runner stores artifacts (logs, facts, etc.) Returns: dict with keys: success (bool), stdout, stderr, status, rc """ playbook_path = os.path.join(ANSIBLE_PLAYBOOK_DIR, playbook_name) if not os.path.exists(playbook_path): return { "success": False, "error": f"Playbook not found: {playbook_path}", "stdout": "", "stderr": "", } logger.info(f" Running playbook: {playbook_name}") logger.info(f" Variables: {json.dumps(extra_vars, indent=2)}") os.makedirs(private_data_dir, exist_ok=True) ## ansible_runner.run() is synchronous -- it blocks until the playbook finishes ## For production, use ansible_runner.run_async() and poll for status result = ansible_runner.run( private_data_dir=private_data_dir, playbook=playbook_path, inventory=ANSIBLE_INVENTORY, extravars=extra_vars, quiet=False, ## set True in production to reduce log noise ) ## ansible-runner returns status: "successful", "failed", "canceled", "timeout" success = result.status == "successful" ## Collect stdout from all events stdout_lines = [] for event in result.events: if event.get("event") in ("runner_on_ok", "runner_on_failed", "debug"): stdout = event.get("event_data", {}).get("res", {}).get("msg", "") if stdout: stdout_lines.append(stdout) if success: logger.info(f" ✅ Playbook succeeded: {playbook_name}") else: logger.error(f" ❌ Playbook failed: {playbook_name} (status: {result.status})") return { "success": success, "status": result.status, "rc": result.rc, "stdout": "\n".join(stdout_lines), "stderr": result.stderr.read() if result.stderr else "", "artifacts_dir": private_data_dir, } ## ── Action-to-playbook mapping ──────────────────────────────── ## Maps action names (from the LLM decision) to playbook filenames ## and required variables. ACTION_PLAYBOOK_MAP = { "restart_pod": { "playbook": "restart_pod.yaml", "required_vars": ["namespace", "pod_name", "service_name"], }, "scale_up_deployment": { "playbook": "scale_up_deployment.yaml", "required_vars": ["namespace", "deployment_name"], }, "increase_memory_limit": { "playbook": "increase_memory_limit.yaml", "required_vars": ["namespace", "deployment_name", "container_name", "new_memory_limit"], }, "rollback_deployment": { "playbook": "rollback_deployment.yaml", "required_vars": ["namespace", "deployment_name"], }, } def execute_action(action_name, action_vars): """ Execute a named remediation action with the given variables. Parameters: action_name: one of the keys in ACTION_PLAYBOOK_MAP action_vars: dict of variables the playbook needs Returns: dict with execution result """ if action_name not in ACTION_PLAYBOOK_MAP: return { "success": False, "error": f"Unknown action: {action_name}. Known actions: {list(ACTION_PLAYBOOK_MAP.keys())}", } mapping = ACTION_PLAYBOOK_MAP[action_name] ## Verify all required variables are present missing = [v for v in mapping["required_vars"] if v not in action_vars] if missing: return { "success": False, "error": f"Missing required variables for {action_name}: {missing}", } return run_playbook(mapping["playbook"], action_vars) ```
What you are building and why it matters Swiggy processes roughly 1.5 million orders a day. During peak dinner hours, ev...
How all three capstones connect This is the architecture you are building. Read it carefully before touching any code. ┌...
Setting up from the capstone 1 and 2 directories aiops-self-healing/ src/ config.py <- all settings including autonomy t...
The config file controls what the system is allowed to do > 📌 Remember: The AUTONOMOUSACTIONS list is your blast radius...
Why Ansible instead of running kubectl directly You could call subprocess.run(["kubectl", "rollout", "restart", ...]) di...
Running playbooks safely from Python...
The gate between diagnosis and action...
Every action needs a permanent record In any automated system that can modify production infrastructure, every action mu...
Confirming the fix actually worked...
Getting human approval for Tier 3 actions curl -X POST http://localhost:8081/approve -d '{{"action": "{actionname}", "se...
The loop that connects everything...
Run the complete self-healing demo Verify the audit log Test a Tier 3 action manually...
...
...
Adding irreversible actions to the autonomous tier. The most dangerous mistake you can make with this system is putting ...
Component File What it Does Anomaly detection Capstone 1 model Runs every 60s, flags unusual metrics RAG search Capstone...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.