Build a Self-Healing Infrastructure System

How all three capstones connect

This is the architecture you are building. Read it carefully before touching any code.

The Decision Engine is the most important new piece. Without it, you have a system that might restart pods, scale deployments, or flush caches based on a wrong diagnosis. The decision engine defines what the system is allowed to do on its own — and forces human approval for anything riskier.

The three tiers of autonomy

Not every remediation action carries the same risk. The system organises actions into three tiers:

Tier 1 — Fully autonomous (system acts immediately): Pod restart, cache flush, horizontal scale-up. These are reversible, low-risk, and run hundreds of times a year. Automating them saves the most on-call time with the least risk.

Tier 2 — Autonomous with notification (system acts, then tells humans): Memory limit increase, rolling restart of a deployment, draining a node. Higher impact but still reversible. The system acts immediately and posts the result to Slack so engineers can review.

Tier 3 — Human approval required (system asks before acting): Rollback of a recent deployment, scaling down a cluster, any action involving databases or persistent storage. These actions can cause data loss or downtime if wrong. A human must approve via Slack button before the system proceeds.

Setting up from the capstone 1 and 2 directories

The config file controls what the system is allowed to do

Remember

The AUTONOMOUS_ACTIONS list is your blast radius control. Start it small — just restart_pod — and expand it as you build confidence in the system's accuracy. Never add irreversible actions to this list.

Why Ansible instead of running kubectl directly

You could call subprocess.run(["kubectl", "rollout", "restart", ...]) directly from Python. Many systems do. The problem is that direct subprocess calls have no idempotency guarantees, no retry logic, no structured output, and no native audit trail.

Ansible gives you all four. Each playbook is idempotent (safe to run twice), has built-in retry with backoff, produces structured JSON output you can parse, and Ansible Runner writes every execution to a persistent artifact directory.

Think of it this way: kubectl is the screwdriver. Ansible is the procedure that says "use the screwdriver, turn it this many times, verify the screw is seated, mark it done."

Running playbooks safely from Python

The gate between diagnosis and action

Every action needs a permanent record

In any automated system that can modify production infrastructure, every action must be auditable. Compliance frameworks like SOC 2 require it. Post-incident investigations depend on it. And if the system ever makes a wrong call, you need to know exactly what it did and why.

Confirming the fix actually worked

Getting human approval for Tier 3 actions

curl -X POST http://localhost:8081/approve -d '{{"action": "{action_name}", "service": "{service_name}", "approved": true}}'

curl -X POST http://localhost:8081/approve -d '{{"action": "{action_name}", "service": "{service_name}", "approved": false}}'

The loop that connects everything

Run the complete self-healing demo

Verify the audit log

Test a Tier 3 action manually

Adding irreversible actions to the autonomous tier. The most dangerous mistake you can make with this system is putting rollback_deployment or scale_down_deployment in AUTONOMOUS_ACTIONS. A wrong diagnosis combined with autonomous rollback can take a working deployment down at 2 AM and cause more damage than the original anomaly. The rule is: if the action cannot be undone in 60 seconds, it goes in APPROVAL_REQUIRED_ACTIONS. Always.

Not setting a rate limit and creating an action loop. A broken pod that keeps crashing will keep triggering the detection model. Without MAX_ACTIONS_PER_SERVICE_PER_HOUR, the system will restart it again and again in an infinite loop. Three restarts in an hour is usually the right limit. If the same service needs more than three restarts in an hour, the underlying cause is not something a restart fixes — a human needs to investigate.

Running two replicas of the orchestrator. With replicas: 2, both pods detect the same anomaly and both start the remediation pipeline simultaneously. You get two Ansible playbook executions, two scale-up operations, and two Slack messages. Keep this at exactly one replica and use a liveness probe to handle pod failures.

Skipping the verification step during testing. The verification step waits 90 seconds and then checks whether the problem resolved. This is slow during development. Developers often set VERIFICATION_WAIT_SECONDS = 0 to speed up testing and then forget to restore it for production. If verification is skipped, the system never learns when its remediations fail — it will report success for actions that did nothing. Keep the wait at 90 seconds in production minimum.

Training the detection model during a past incident. The Isolation Forest model from Capstone 1 learns what "normal" looks like from the training data. If you train it on a 24-hour window that included an incident, it learns that incident conditions are normal. The self-healing system will never trigger during the same type of incident because the model thinks it is expected behaviour. Before training, check your monitoring dashboards to confirm the training window was genuinely incident-free.

Not testing the approval workflow before it matters. The Tier 3 approval flow goes untested until a real production incident requires a rollback. Then it fails because the Slack bot token expired, or the approval endpoint is unreachable, or nobody knows the curl command format. Run a monthly drill: force a Tier 3 action and verify the approval flow works end to end.