Prometheus Anomaly Detection with Isolation Forest

Build a Prometheus anomaly detector in Python: train Isolation Forest on cluster metrics, alert via Alertmanager, and deploy it on Kubernetes.

Architecture Overview & Problem Statement

This project builds a Prometheus anomaly detector in Python. It collects Kubernetes cluster metrics, trains an Isolation Forest model on normal behaviour, scores live metrics every 60 seconds, and sends an alert through Alertmanager when behaviour looks unusual, even if no fixed threshold has been crossed.

The problem it solves

Imagine you are on call for a food-delivery app like Swiggy during the dinner rush. The order service is getting slower: responses that normally take 50ms now take 180ms. Nothing is crashing and the error rate is still 0%, so a threshold alert such as "error rate above 5%" never fires. Users still notice, and complaints start.

Anomaly detection covers this gap. Instead of hand-writing a threshold for every failure mode, you train a model on what normal looks like. When several metrics together look different from normal, the model flags it. Think of a security guard who has memorised what the building looks like during normal hours and notices when something is off. The Isolation Forest is that guard.

How the pieces fit

Training runs once. It turns 24 hours of Prometheus history into a saved model:

flowchart TD A[Prometheus] -->|query_range| B[collect_metrics.py] B --> C[metrics.csv] C --> D[train_model.py] D --> E[Model and scaler] classDef blue fill:#326CE5,stroke:#1E3A8A,color:#ffffff classDef purple fill:#8B5CF6,stroke:#5B21B6,color:#ffffff classDef orange fill:#F97316,stroke:#C2410C,color:#ffffff classDef green fill:#10B981,stroke:#047857,color:#ffffff classDef red fill:#EF4444,stroke:#B91C1C,color:#ffffff classDef slate fill:#475569,stroke:#1E293B,color:#ffffff class A blue class B,D purple class C orange class E green

Detection runs continuously. Every 60 seconds it loads the model, scores the current metrics, and logs the result. Normal scores just continue the loop. It alerts only after 3 anomalous checks in a row:

flowchart TD A[Prometheus] -->|current values| B[detect_anomalies.py] B -->|3 anomalous checks| C[Alertmanager] C --> D[Slack or PagerDuty] classDef blue fill:#326CE5,stroke:#1E3A8A,color:#ffffff classDef purple fill:#8B5CF6,stroke:#5B21B6,color:#ffffff classDef orange fill:#F97316,stroke:#C2410C,color:#ffffff classDef green fill:#10B981,stroke:#047857,color:#ffffff classDef red fill:#EF4444,stroke:#B91C1C,color:#ffffff classDef slate fill:#475569,stroke:#1E293B,color:#ffffff class A blue class B purple class C red class D slate

What you will build

A Python collector that pulls metrics from Prometheus
A training script that fits an Isolation Forest on a window of normal behaviour
A detector that runs every 60 seconds and scores the current cluster state
An alert module that sends anomalies to Alertmanager (API v2), with keep-alive and recovery
A Dockerfile and Kubernetes Deployment that run the detector in the cluster

What you will learn

How to query the Prometheus range and instant APIs from Python
How Isolation Forest works, and why feature scaling and the training window matter
How to turn a model score into an alert without causing alert storms
How Alertmanager treats alerts (fingerprints, endsAt, auto-resolve)
How to measure a detector instead of trusting it

What you can show in your portfolio

A GitHub repo with the full pipeline and a README
An architecture diagram and the training chart (data/training_visualisation.png)
A screenshot of the AnomalyDetected alert in Alertmanager
A results table with your measured time-to-detect and false-alert count

Before you start

This is the first of two AIOps projects. The second, the AI Incident Response Agent project, diagnoses alerts like the ones you create here. You can do either one on its own. Allow 3 to 4 hours, plus at least 1 hour of detector run time to measure false alerts.

You need:

Python 3.9 to 3.11 (the pinned library versions below do not support 3.12)
A running Kubernetes cluster (minikube works)
Prometheus and Alertmanager installed on the cluster (from the Monitoring and Logging module)
Basic Python knowledge (from the Python for Ops module)

Bash

## Verify your setup before starting
python3 --version
## Should show Python 3.9, 3.10 or 3.11
 
kubectl get pods -n monitoring | grep -E "prometheus|alertmanager"
## Should show Prometheus and Alertmanager pods Running
 
echo "✅ Ready to build"

An honest note on limits

Anomaly detection finds behaviour that is different, not behaviour that is bad. A planned traffic spike can be flagged, and a slow degradation can be missed if it was already in your training data. Use this alongside normal threshold alerts, not instead of them. The last milestones have you measure how well it works on your own cluster, because the numbers depend on your cluster.

What Is Prometheus and Why Are We Using It?

Prometheus collects metrics from your services and stores them as time-series data: a value recorded at a point in time, repeated over and over.

◈ DIAGRAM

cpu_usage{pod="backend-abc"} = 23%  at 14:00:00
cpu_usage{pod="backend-abc"} = 25%  at 14:00:15
cpu_usage{pod="backend-abc"} = 89%  at 14:00:30  ← something changed here
cpu_usage{pod="backend-abc"} = 91%  at 14:00:45

Prometheus has an HTTP API for both current values (/api/v1/query) and history (/api/v1/query_range). We use the history to train the model and the current value to run detection.

What Is Isolation Forest and Why Are We Using It?

Isolation Forest is an algorithm built for anomaly detection. The intuition:

Imagine 1000 points representing normal CPU usage between 20% and 40%, plus one point at 95%. If you draw random boundaries to separate points from each other, the 95% point is alone, so very few boundaries isolate it. Normal points are packed together and need many boundaries.

Isolation Forest builds many random trees and measures how quickly each point gets isolated. Fast isolation means a more anomalous point.

Why it suits this project:

It needs no labelled incidents. You only need data from a period when things were normal.
It handles several metrics at once, so it can notice an unusual combination of values.
It is fast to train and fast to score.

How scoring works (read this once, it matters later)

scikit-learn gives you two numbers for each point:

score_samples: the raw anomaly score (lower means more anomalous)
decision_function: the raw score shifted by a threshold learned from the contamination setting. Below 0 means anomaly, above 0 means normal.

We use decision_function for alerting, because the cut-off is always 0 no matter what the raw scores look like. The contamination setting decides where that cut-off sits: it is the fraction of training data the model will label as anomalous. A higher contamination means a looser cut-off and more alerts. A lower value means a stricter cut-off and fewer alerts.

What Is Alertmanager?

Alertmanager receives alerts and routes them to Slack, email, PagerDuty or a webhook. Two behaviours matter for this project:

Alerts are identified by their labels. Sending the same labels again updates the same alert instead of creating a new one.
An alert is considered resolved when its endsAt time passes. If you send an alert once with no endsAt, Alertmanager resolves it by itself after its resolve_timeout (5 minutes by default), even if the problem continues. So a detector has to keep re-sending an alert while the anomaly lasts.

Milestone 1: Set Up the Project and Configuration

This milestone creates the project layout, installs pinned Python dependencies, and puts every setting for the detector in one config.py file, so changing Prometheus URLs, queries or thresholds later means editing one place.

Bash

## Create the project directory
mkdir aiops-anomaly-detection && cd aiops-anomaly-detection
 
mkdir -p src data models logs k8s
 
touch src/__init__.py
touch src/config.py
touch src/collect_metrics.py
touch src/train_model.py
touch src/alert.py
touch src/detect_anomalies.py
touch requirements.txt
 
echo "✅ Project structure created"

◈ DIAGRAM

aiops-anomaly-detection/
  src/
    config.py            ← all settings in one place
    collect_metrics.py   ← pulls data from Prometheus
    train_model.py       ← trains the Isolation Forest model
    alert.py             ← sends alerts to Alertmanager
    detect_anomalies.py  ← runs detection every 60 seconds
  data/                  ← collected metrics and charts
  models/                ← trained model
  logs/                  ← detection logs
  k8s/                   ← Kubernetes manifests
  requirements.txt

Requirements File

Bash

# requirements.txt
pandas==2.1.0
scikit-learn==1.3.0
numpy==1.24.3
requests==2.31.0
matplotlib==3.7.2
joblib==1.3.2

Bash

## Use a virtual environment so these pins do not clash with other projects
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
echo "✅ Dependencies installed"

Why Configuration Belongs in One File

The Prometheus URL, the metric queries and the alert thresholds live in one place. When you point the system at a different cluster or add a metric, you change one file.

Python

# src/config.py
# All settings for the anomaly detection pipeline.
# Change these values to match your environment.
 
import os
 
# ── Prometheus Connection ────────────────────────────────────
# For minikube run:
#   kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090
# then use http://localhost:9090
PROMETHEUS_URL = os.getenv("PROMETHEUS_URL", "http://localhost:9090")
 
# ── Metrics to Collect ───────────────────────────────────────
# Each query returns ONE number for the whole cluster, so the model sees
# cluster-wide patterns. The same queries are used for training and detection,
# which is important: the model must see the same kind of number in both.
METRICS = {
    # Average CPU usage per container, in cores
    "cpu_usage": 'avg(rate(container_cpu_usage_seconds_total{container!=""}[5m]))',
 
    # Average memory working set per container, in bytes
    "memory_usage": 'avg(container_memory_working_set_bytes{container!=""})',
 
    # Requests per second across all services
    "http_request_rate": 'sum(rate(http_requests_total[5m]))',
 
    # Fraction of requests returning 5xx.
    # "or vector(0)" matters: when there are no 5xx at all, Prometheus returns
    # NO series, and the metric would be dropped from training.
    "http_error_rate": '(sum(rate(http_requests_total{status=~"5.."}[5m])) or vector(0)) / sum(rate(http_requests_total[5m]))',
 
    # 95th percentile response time in seconds
    "response_time_p95": 'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))',
}
 
# ── Data Collection ──────────────────────────────────────────
# Hours of history used for training. At a 60s step, 24 hours = 1440 points.
TRAINING_DATA_HOURS = 24
TRAINING_STEP = "60s"
 
# How often the detector checks the cluster, in seconds
DETECTION_INTERVAL_SECONDS = 60
 
# ── Model Settings ───────────────────────────────────────────
# Contamination = the fraction of TRAINING data the model labels as anomalous.
# It sets how strict the detector is:
#   higher value -> looser cut-off -> MORE alerts
#   lower value  -> stricter cut-off -> FEWER alerts
# Start low (0.01) and raise it only if you are missing real anomalies.
CONTAMINATION = 0.01
 
# Alert only after this many anomalous checks in a row (reduces false alerts
# from short spikes). With a 60s interval, 3 means 3 minutes.
CONSECUTIVE_ANOMALIES_REQUIRED = 3
 
MODEL_PATH = "models/isolation_forest.joblib"
DATA_PATH = "data/metrics.csv"
 
# ── Alertmanager Connection ──────────────────────────────────
ALERTMANAGER_URL = os.getenv("ALERTMANAGER_URL", "http://localhost:9093")
ALERT_ENVIRONMENT = os.getenv("ALERT_ENVIRONMENT", "lab")
 
# While an anomaly lasts, the alert is re-sent every cycle with an endsAt this
# many seconds in the future, so Alertmanager keeps it firing.
ALERT_TTL_SECONDS = DETECTION_INTERVAL_SECONDS * 3
 
# ── Logging ──────────────────────────────────────────────────
LOG_PATH = "logs/detection.log"

Common Mistake
Installing on Python 3.12 fails because numpy==1.24.3 has no build for it. Use Python 3.9 to 3.11 and a virtual environment, so these pins do not clash with other projects.

Milestone 2: Collect Metrics from Prometheus

This milestone pulls 24 hours of cluster metrics from the Prometheus HTTP API and saves them as a CSV training set, one row per minute and one column per metric.

What This File Does

It connects to Prometheus, runs the queries and saves the results to a CSV file (one row per timestamp, one column per metric).

Why save to CSV first? Collecting 24 hours of data takes a moment. You collect once, save it, and train from the file. If training fails you do not have to collect again, and you can open the CSV to inspect the data.

Python

# src/collect_metrics.py
# Collects metrics from Prometheus for training and for live detection.
 
import math
import requests
import pandas as pd
from datetime import datetime, timedelta
from src.config import (
    PROMETHEUS_URL, METRICS, TRAINING_DATA_HOURS, TRAINING_STEP, DATA_PATH
)
 
 
def query_prometheus_range(metric_name, query, start_time, end_time, step):
    """
    Query Prometheus for history over a time range.
 
    Returns a list of (timestamp, value) tuples, or an empty list on failure.
    NaN and infinite values are dropped, because Prometheus returns "NaN"
    for things like 0/0 and the model cannot train on them.
    """
    url = f"{PROMETHEUS_URL}/api/v1/query_range"
    params = {
        "query": query,
        "start": start_time.timestamp(),
        "end": end_time.timestamp(),
        "step": step,
    }
 
    try:
        response = requests.get(url, params=params, timeout=30)
        response.raise_for_status()
        data = response.json()
 
        if data["status"] != "success":
            print(f"⚠️  Query failed for {metric_name}: {data.get('error', 'unknown error')}")
            return []
 
        results = data["data"]["result"]
        if not results:
            print(f"⚠️  No data for {metric_name}. Does this metric exist in your cluster?")
            return []
 
        # Our queries use avg/sum, so there is exactly one series
        values = []
        for ts, val in results[0]["values"]:
            number = float(val)
            if math.isfinite(number):
                values.append((float(ts), number))
        return values
 
    except requests.exceptions.ConnectionError:
        print(f"❌ Cannot connect to Prometheus at {PROMETHEUS_URL}")
        print("   Run: kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090")
        return []
    except Exception as e:
        print(f"❌ Error querying {metric_name}: {e}")
        return []
 
 
def collect_training_data():
    """Collect TRAINING_DATA_HOURS of metrics and save them to DATA_PATH."""
    print(f"📊 Collecting {TRAINING_DATA_HOURS} hours of metrics from Prometheus...")
    print(f"   URL: {PROMETHEUS_URL}")
    print(f"   Metrics: {list(METRICS.keys())}\n")
 
    end_time = datetime.now()
    start_time = end_time - timedelta(hours=TRAINING_DATA_HOURS)
    print(f"   Time range: {start_time:%Y-%m-%d %H:%M} → {end_time:%Y-%m-%d %H:%M}\n")
 
    all_data = {}
    for metric_name, query in METRICS.items():
        print(f"   Collecting {metric_name}...", end=" ")
        values = query_prometheus_range(metric_name, query, start_time, end_time, TRAINING_STEP)
        if values:
            all_data[metric_name] = {ts: val for ts, val in values}
            print(f"✅ {len(values)} data points")
        else:
            print("⚠️  Skipped (no data)")
 
    if not all_data:
        print("\n❌ No metrics collected. Check the Prometheus connection and metric names.")
        return False
 
    # Keep only timestamps present in every metric, so every row is complete.
    # All queries share the same start, end and step, so timestamps line up.
    common = sorted(set.intersection(*[set(d.keys()) for d in all_data.values()]))
    print(f"\n   Aligning {len(common)} common timestamps across {len(all_data)} metrics...")
 
    rows = []
    for ts in common:
        row = {"timestamp": datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")}
        for metric_name in all_data:
            row[metric_name] = all_data[metric_name][ts]
        rows.append(row)
 
    df = pd.DataFrame(rows)
 
    if len(df) < 300:
        print(f"\n⚠️  Only {len(df)} rows. Training on this little data gives a weak model.")
        print("   Aim for at least 300 rows (about 5 hours at a 60s step).")
 
    df.to_csv(DATA_PATH, index=False)
    print(f"\n✅ Saved {len(df)} rows to {DATA_PATH}")
    print(f"   Columns: {list(df.columns)}")
    print("\n   Preview:")
    print(df.head(3).to_string())
    return True
 
 
def collect_single_point(metric_names=None):
    """
    Collect the current value of each metric, for live detection.
 
    Returns {metric_name: value}, or None if ANY metric is missing or NaN.
    We skip the cycle instead of filling a missing metric with 0, because a
    fake 0 would look like a real anomaly to the model.
    """
    names = metric_names or list(METRICS.keys())
    current = {}
 
    for name in names:
        try:
            response = requests.get(
                f"{PROMETHEUS_URL}/api/v1/query",
                params={"query": METRICS[name]},
                timeout=10,
            )
            data = response.json()
            if data["status"] != "success" or not data["data"]["result"]:
                print(f"⚠️  No current data for {name}")
                return None
 
            value = float(data["data"]["result"][0]["value"][1])
            if not math.isfinite(value):
                print(f"⚠️  {name} is NaN right now (no traffic?)")
                return None
            current[name] = value
 
        except Exception as e:
            print(f"⚠️  Could not query {name}: {e}")
            return None
 
    return current
 
 
if __name__ == "__main__":
    collect_training_data()

Bash

## Port-forward Prometheus (leave this running in another terminal)
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 &
 
## Collect the training data
python3 -m src.collect_metrics
 
## Expected output (numbers depend on your cluster):
## 📊 Collecting 24 hours of metrics from Prometheus...
##    Collecting cpu_usage... ✅ 1441 data points
##    Collecting memory_usage... ✅ 1441 data points
##    ...
## ✅ Saved 1441 rows to data/metrics.csv
 
head -5 data/metrics.csv

If a metric is skipped. Metrics like http_requests_total only exist if your applications expose them. A skipped metric is fine: the model is trained on the columns that exist, and the detector only queries those columns. Having at least cpu and memory is enough to complete the project. Adding app-level metrics makes it much more useful.

Common Mistake
Training on a window that contains a past incident. If last Tuesday's outage is in your 24 hours of data, the model learns that very high CPU is normal and stays quiet during the next incident. Check Grafana first, pick a genuinely quiet window, and note the date range in your README.

Common Mistake
Monitoring only CPU and memory. When a database connection pool fills up, response times jump from 50ms to 8 seconds while CPU and memory look normal. Include application metrics (error rate, latency percentiles, request rate) so the model can see failures that infrastructure metrics miss.

Milestone 3: Train the Isolation Forest Model

This milestone trains an Isolation Forest on the CSV, scales the features, and saves the model and scaler together in one joblib file that the detector loads later.

What Is Happening Here

Training lets the Isolation Forest look at all your collected rows and learn the shape of normal. You do not label anything.

Two preparation steps matter:

Scaling. Memory is in billions of bytes and CPU is between 0 and 1. Without scaling the model would treat the large numbers as far more important. StandardScaler converts each column to (value - mean) / standard deviation.
Saving the scaler with the model. At detection time, live data must be scaled with the training mean and deviation. We save the scaler and model together in one file so they cannot get out of sync.

Python

# src/train_model.py
# Trains an Isolation Forest on the collected metrics and saves it.
 
import os
import joblib
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from src.config import DATA_PATH, MODEL_PATH, CONTAMINATION
 
 
def load_and_prepare_data():
    """Load the CSV and scale the features. Returns (df, X_scaled, scaler, columns)."""
    if not os.path.exists(DATA_PATH):
        print(f"❌ Data file not found at {DATA_PATH}")
        print("   Run: python3 -m src.collect_metrics")
        return None, None, None, None
 
    print(f"📂 Loading data from {DATA_PATH}...")
    df = pd.read_csv(DATA_PATH)
    print(f"   Shape: {df.shape[0]} rows × {df.shape[1]} columns")
 
    # The timestamp identifies a row but is not a feature
    feature_columns = [c for c in df.columns if c != "timestamp"]
    X = df[feature_columns].copy()
 
    # Fill any leftover gaps with the median (a spike would distort the mean)
    for col in X.columns:
        if X[col].isnull().any():
            X[col] = X[col].fillna(X[col].median())
            print(f"   Filled missing values in {col} with the median")
 
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X)
 
    print("\n   Training data summary (before scaling):")
    for col in feature_columns:
        print(f"   {col}: mean={df[col].mean():.6g}, std={df[col].std():.6g}")
 
    return df, X_scaled, scaler, feature_columns
 
 
def train_and_save_model(X_scaled, scaler, feature_columns):
    """Train the Isolation Forest and save it together with the scaler."""
    print("\n🤖 Training Isolation Forest...")
    print(f"   Contamination: {CONTAMINATION} (about {CONTAMINATION * 100:.1f}% of training rows will be labelled anomalous)")
    print(f"   Samples: {X_scaled.shape[0]}, features: {X_scaled.shape[1]}")
 
    # n_estimators=100: number of random trees
    # random_state=42: same data gives the same model every time
    model = IsolationForest(
        n_estimators=100,
        contamination=CONTAMINATION,
        random_state=42,
        n_jobs=-1,
    )
    model.fit(X_scaled)
 
    # decision_function: below 0 = anomaly, above 0 = normal
    scores = model.decision_function(X_scaled)
    flagged = scores < 0
 
    print("\n   Training results:")
    print(f"   Normal rows:  {(~flagged).sum()} ({(~flagged).mean() * 100:.1f}%)")
    print(f"   Flagged rows: {flagged.sum()} ({flagged.mean() * 100:.1f}%)")
    print(f"   Score range:  {scores.min():.4f} to {scores.max():.4f} (below 0 = anomalous)")
 
    package = {
        "model": model,
        "scaler": scaler,
        "feature_columns": feature_columns,
        "training_samples": int(X_scaled.shape[0]),
        "contamination": CONTAMINATION,
    }
    os.makedirs(os.path.dirname(MODEL_PATH), exist_ok=True)
    joblib.dump(package, MODEL_PATH)
 
    print(f"\n✅ Model saved to {MODEL_PATH} ({os.path.getsize(MODEL_PATH) / 1024:.1f} KB)")
    return flagged
 
 
def visualise_training_data(df, flagged):
    """Plot each metric over time and mark the rows the model flagged in red."""
    try:
        import matplotlib
        matplotlib.use("Agg")   # no display needed
        import matplotlib.pyplot as plt
 
        feature_cols = [c for c in df.columns if c != "timestamp"]
        ts = pd.to_datetime(df["timestamp"])
 
        fig, axes = plt.subplots(
            len(feature_cols), 1, figsize=(14, 3 * len(feature_cols)),
            sharex=True, squeeze=False,
        )
        fig.suptitle("Training data: metrics over time (red = flagged by the model)",
                     fontsize=14, fontweight="bold")
 
        for ax, col in zip(axes[:, 0], feature_cols):
            ax.plot(ts, df[col], linewidth=0.8, color="#F59E0B")
            ax.scatter(ts[flagged], df.loc[flagged, col], color="#DC2626", s=14)
            ax.set_ylabel(col, fontsize=9)
            ax.grid(True, alpha=0.3)
 
        plt.tight_layout()
        plt.savefig("data/training_visualisation.png", dpi=150, bbox_inches="tight")
        print("📊 Chart saved to data/training_visualisation.png")
 
    except Exception as e:
        print(f"⚠️  Could not create chart: {e}")
 
 
if __name__ == "__main__":
    df, X_scaled, scaler, feature_columns = load_and_prepare_data()
    if X_scaled is not None:
        flagged = train_and_save_model(X_scaled, scaler, feature_columns)
        visualise_training_data(df, flagged)
        print("\n🎯 Model ready. Next: python3 -m src.detect_anomalies")

Bash

python3 -m src.train_model
 
## Expected output (numbers depend on your data):
## 📂 Loading data from data/metrics.csv...
##    Shape: 1441 rows × 6 columns
## 🤖 Training Isolation Forest...
##    Flagged rows: 15 (1.0%)
## ✅ Model saved to models/isolation_forest.joblib
 
ls -lh models/ data/

Open data/training_visualisation.png. Look at the red points. They should be the odd moments in your data. If the chart shows a long period of very high CPU that you recognise as an incident, collect training data from a different window. A model trained on an incident treats that incident as normal.

Tip
Open data/training_visualisation.png and look at the red points. They should be odd moments you recognise. If a whole incident is not flagged, your training window is contaminated.

Common Mistake
Setting CONTAMINATION too high. At 0.10 the model labels about 10% of normal behaviour as anomalous by design, so engineers get alert after alert for ordinary variation and start ignoring the channel. A higher value means more alerts. Start at 0.01 and raise it slowly only if you miss real anomalies.

Milestone 4: Send Alerts to Alertmanager

This milestone sends anomaly alerts to the Alertmanager v2 API, keeps them firing while the anomaly lasts, and resolves them when metrics return to normal.

Why a Separate Alert Module

This file has one job: send alerts to Alertmanager. If you later want Slack directly, you only change this file.

Three details are easy to get wrong, and this module handles each:

Use API v2. Alertmanager removed the old /api/v1/alerts endpoint in version 0.27. Use /api/v2/alerts.
Use real UTC timestamps. Alertmanager expects RFC 3339 times. We use timezone-aware UTC datetimes.
Send endsAt. Without it the alert auto-resolves after the resolve_timeout. We set endsAt a few minutes ahead and re-send each cycle while the anomaly lasts.

Python

# src/alert.py
# Sends anomaly alerts to Alertmanager (API v2).
 
import requests
from datetime import datetime, timedelta, timezone
from src.config import ALERTMANAGER_URL, ALERT_ENVIRONMENT, ALERT_TTL_SECONDS
 
 
def _labels():
    # Labels identify the alert. Same labels = same alert (updated, not duplicated).
    return {
        "alertname": "AnomalyDetected",
        "severity": "warning",
        "source": "aiops-anomaly-detector",
        "environment": ALERT_ENVIRONMENT,
    }
 
 
def _post(alerts):
    try:
        response = requests.post(
            f"{ALERTMANAGER_URL}/api/v2/alerts",
            json=alerts,
            timeout=10,
        )
        if response.status_code == 200:
            return True
        print(f"   ⚠️  Alertmanager returned {response.status_code}: {response.text}")
        return False
    except requests.exceptions.ConnectionError:
        print(f"   ⚠️  Cannot reach Alertmanager at {ALERTMANAGER_URL}")
        print("   Run: kubectl port-forward -n monitoring svc/alertmanager-operated 9093:9093")
        return False
    except Exception as e:
        print(f"   ⚠️  Alert failed: {e}")
        return False
 
 
def _summary(metric_values):
    return " | ".join(f"{k}: {v:.4g}" for k, v in metric_values.items())
 
 
def send_anomaly_alert(decision_score, metric_values, started_at):
    """
    Send (or refresh) the anomaly alert.
 
    decision_score: the model's decision_function value (below 0 = anomalous)
    started_at: timezone-aware datetime when this anomaly first triggered
    The alert expires ALERT_TTL_SECONDS from now unless it is re-sent.
    """
    now = datetime.now(timezone.utc)
    alert = [{
        "labels": _labels(),
        "annotations": {
            "summary": "Anomaly detected in cluster metrics",
            "description": (
                f"The anomaly detection model flagged unusual behaviour. "
                f"Decision score: {decision_score:.4f} (below 0 is anomalous). "
                f"Current metrics: {_summary(metric_values)}"
            ),
        },
        "startsAt": started_at.isoformat(),
        "endsAt": (now + timedelta(seconds=ALERT_TTL_SECONDS)).isoformat(),
    }]
    ok = _post(alert)
    if ok:
        print("   🔔 Alert sent to Alertmanager")
    return ok
 
 
def send_recovery_alert(started_at):
    """Resolve the alert right away by sending it with endsAt = now."""
    now = datetime.now(timezone.utc)
    alert = [{
        "labels": _labels(),
        "annotations": {"summary": "Anomaly resolved: metrics returned to normal"},
        "startsAt": started_at.isoformat(),
        "endsAt": now.isoformat(),
    }]
    if _post(alert):
        print("   ✅ Recovery sent")

You can test this module before the detector exists:

Bash

## Port-forward Alertmanager (leave running)
kubectl port-forward -n monitoring svc/alertmanager-operated 9093:9093 &
 
python3 -c "
from datetime import datetime, timezone
from src.alert import send_anomaly_alert
send_anomaly_alert(-0.05, {'cpu_usage': 0.9}, datetime.now(timezone.utc))
"
## Open http://localhost:9093 and look for AnomalyDetected.
## It disappears by itself in about 3 minutes, because nothing keeps it alive.

Common Mistake
Copying an old tutorial that posts to /api/v1/alerts. Current Alertmanager versions return 404, and the detector looks healthy in its own logs while no alert ever exists. Use /api/v2/alerts and confirm in the Alertmanager UI.

Common Mistake
Sending the alert once with no endsAt. Alertmanager auto-resolves it after about 5 minutes even if the problem continues. Send an endsAt a few minutes ahead and re-send every cycle while the anomaly lasts.

Milestone 5: Run Continuous Detection

This milestone runs the detector loop: every 60 seconds it scores the current metrics with the model, waits for 3 anomalous checks in a row, then alerts and later sends a recovery.

What This File Does

Every 60 seconds it:

Collects the current value of each metric the model was trained on
Scales it with the training scaler
Gets the decision score from the model
Counts consecutive anomalous checks. After 3 in a row it sends an alert, and re-sends it every cycle while the anomaly lasts
When metrics return to normal, sends a recovery

Python

# src/detect_anomalies.py
# The main detection loop.
 
import os
import time
import logging
import joblib
import pandas as pd
from datetime import datetime, timezone
from src.config import (
    MODEL_PATH, DETECTION_INTERVAL_SECONDS, LOG_PATH,
    CONSECUTIVE_ANOMALIES_REQUIRED, PROMETHEUS_URL, ALERTMANAGER_URL,
)
from src.collect_metrics import collect_single_point
from src.alert import send_anomaly_alert, send_recovery_alert
 
os.makedirs(os.path.dirname(LOG_PATH), exist_ok=True)
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.FileHandler(LOG_PATH), logging.StreamHandler()],
)
logger = logging.getLogger(__name__)
 
 
def load_model():
    """Load the model package. Exits if you have not trained yet."""
    if not os.path.exists(MODEL_PATH):
        logger.error(f"Model not found at {MODEL_PATH}. Run: python3 -m src.train_model")
        raise SystemExit(1)
 
    package = joblib.load(MODEL_PATH)
    logger.info(f"Model loaded (trained on {package['training_samples']} samples)")
    logger.info(f"Features: {package['feature_columns']}")
    return package["model"], package["scaler"], package["feature_columns"]
 
 
def detect_once(model, scaler, feature_columns):
    """
    Run one check. Returns (is_anomaly, decision_score, metric_values),
    or (None, None, None) if metrics could not be collected.
    """
    metric_values = collect_single_point(feature_columns)
    if metric_values is None:
        return None, None, None
 
    # Same column order as training. The scaler depends on it.
    X = pd.DataFrame([metric_values])[feature_columns]
    X_scaled = scaler.transform(X)
 
    # decision_function: below 0 = anomaly
    score = float(model.decision_function(X_scaled)[0])
    return score < 0, score, metric_values
 
 
def run_detection_loop():
    model, scaler, feature_columns = load_model()
 
    logger.info("=" * 60)
    logger.info("🚀 Anomaly detection started")
    logger.info(f"   Interval: {DETECTION_INTERVAL_SECONDS}s | alert after {CONSECUTIVE_ANOMALIES_REQUIRED} anomalous checks in a row")
    logger.info(f"   Prometheus: {PROMETHEUS_URL} | Alertmanager: {ALERTMANAGER_URL}")
    logger.info("=" * 60)
 
    consecutive = 0
    alert_started_at = None   # set while an alert is active
 
    while True:
        try:
            is_anomaly, score, metric_values = detect_once(model, scaler, feature_columns)
 
            if is_anomaly is None:
                logger.warning("Metrics unavailable, skipping this cycle")
            else:
                label = "⚠️ ANOMALOUS" if is_anomaly else "✅ NORMAL"
                logger.info(f"Score: {score:+.4f} ({label})")
                for name, value in metric_values.items():
                    logger.info(f"   {name}: {value:.6g}")
 
                if is_anomaly:
                    consecutive += 1
                    logger.warning(f"   Anomalous check {consecutive}/{CONSECUTIVE_ANOMALIES_REQUIRED}")
 
                    if consecutive >= CONSECUTIVE_ANOMALIES_REQUIRED:
                        if alert_started_at is None:
                            alert_started_at = datetime.now(timezone.utc)
                            logger.warning("🚨 ALERT: anomaly confirmed, notifying Alertmanager")
                        # Re-send every cycle so Alertmanager keeps the alert firing
                        send_anomaly_alert(score, metric_values, alert_started_at)
                else:
                    if alert_started_at is not None:
                        logger.info("✅ Back to normal, sending recovery")
                        send_recovery_alert(alert_started_at)
                        alert_started_at = None
                    consecutive = 0
 
        except KeyboardInterrupt:
            logger.info("⛔ Stopped by user")
            break
        except Exception as e:
            logger.error(f"Unexpected error in detection cycle: {e}")
 
        time.sleep(DETECTION_INTERVAL_SECONDS)
 
 
if __name__ == "__main__":
    run_detection_loop()

Bash

python3 -m src.detect_anomalies
 
## Expected output:
## [INFO] Model loaded (trained on 1441 samples)
## [INFO] 🚀 Anomaly detection started
## [INFO] Score: +0.0823 (✅ NORMAL)
## [INFO]    cpu_usage: 0.0234
## ...

Leave this running for at least an hour on a quiet cluster before you inject anything. The number of alerts it sends during that time is your false-alert baseline for Milestone 6.

Common Mistake
Filling a missing metric with 0. A short Prometheus gap becomes a fake drop to zero CPU, and a false alert fires at 3 AM. Skip the cycle when any metric is missing or NaN instead.

Milestone 6: Inject an Anomaly and Measure Detection

This milestone deliberately stresses the cluster and records how long detection takes and how many false alerts you get during normal running.

Why This Step Matters

A detector you have not tested is a guess. Here you cause a real change in the cluster, then measure how long detection takes.

Bash

## Terminal 1: detection must be running (see Milestone 5)
 
## Terminal 2: note the start time, then start a CPU stress pod
date -u +"%H:%M:%S injection started"
 
kubectl run cpu-stress \
  --image=polinux/stress \
  --restart=Never \
  -- stress --cpu 4 --timeout 300s
## 4 busy CPU workers for 5 minutes
 
## Watch Terminal 1. Expect to see:
## ⚠️ Anomalous check 1/3
## ⚠️ Anomalous check 2/3
## ⚠️ Anomalous check 3/3
## 🚨 ALERT: anomaly confirmed, notifying Alertmanager
 
## After the pod finishes, look for the recovery message.
 
kubectl delete pod cpu-stress

Set your expectations honestly.

The CPU metric is a cluster-wide average using a 5-minute rate window, so it moves slowly. Detection can take several minutes, not seconds.
On a large cluster, one stress pod barely moves a cluster-wide average. If nothing is detected, run several stress pods (use a Deployment with replicas: 3) or run it longer. That is a real limitation of cluster-wide metrics, and it is worth writing down in your README.

Record your results

Copy this table into your README and fill it in with your own measurements:

Measurement	Your result
Training window (date range, hours)
Contamination and consecutive-check setting	0.01 and 3
False alerts during 1+ hour of normal running
Time from injection start to alert (minutes)
Time from injection end to recovery (minutes)
Metrics used by the model

Get the numbers from the log:

Bash

grep -E "ALERT|Back to normal" logs/detection.log
grep -c "ANOMALOUS" logs/detection.log

If the false-alert count is too high, lower CONTAMINATION or raise CONSECUTIVE_ANOMALIES_REQUIRED, retrain, and measure again. Writing down what you changed and what happened is the most valuable part of this project.

Tip
A cluster-wide average barely moves when one small pod misbehaves. If nothing is detected, run several stress pods or run them longer, and write that limitation in your README.

Milestone 7: Deploy the Detector to Kubernetes

This milestone packages the detector as a Docker image and runs it as a single-replica Deployment in the monitoring namespace, so detection keeps running when your laptop closes.

Why Deploy to the Cluster?

On your laptop, detection stops when the laptop closes. In the cluster it runs continuously. For simplicity the trained model is baked into the image. Retraining then means rebuilding the image; the last section lists a better setup as a stretch goal.

Dockerfile

## Dockerfile
FROM python:3.11-slim
 
WORKDIR /app
 
## Install dependencies first so Docker caches this layer
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
 
COPY src/ ./src/
COPY models/ ./models/
 
CMD ["python3", "-m", "src.detect_anomalies"]

YAML

## k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: anomaly-detector
  namespace: monitoring
  labels:
    app: anomaly-detector
spec:
  replicas: 1   ## keep at 1: two replicas would send duplicate alerts
  selector:
    matchLabels:
      app: anomaly-detector
  template:
    metadata:
      labels:
        app: anomaly-detector
    spec:
      containers:
      - name: detector
        ## docker build -t your-registry/anomaly-detector:v1.0.0 .
        ## docker push your-registry/anomaly-detector:v1.0.0
        image: your-registry/anomaly-detector:v1.0.0
        env:
        - name: PROMETHEUS_URL
          value: "http://prometheus-operated.monitoring.svc:9090"
        - name: ALERTMANAGER_URL
          value: "http://alertmanager-operated.monitoring.svc:9093"
        - name: ALERT_ENVIRONMENT
          value: "lab"
        resources:
          requests:
            cpu: "100m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Bash

docker build -t your-registry/anomaly-detector:v1.0.0 .
docker push your-registry/anomaly-detector:v1.0.0
 
kubectl apply -f k8s/deployment.yaml
 
kubectl get pods -n monitoring | grep anomaly-detector
kubectl logs -n monitoring deployment/anomaly-detector -f
## Look for "Model loaded" and a first "Score:" line

On minikube you can skip the registry: run eval $(minikube docker-env) before docker build, and set imagePullPolicy: IfNotPresent on the container.

Common Mistake
Never retraining. A model trained in January on 10 pods alerts on healthy behaviour once the cluster has 80. Retrain after major scaling or architecture changes, and at least monthly.

Security
Use a pinned image tag and a private registry, and never bake credentials into the image.

Milestone 8: Package It as a Portfolio Project

This milestone turns the working detector into a portfolio piece with a README, architecture diagram, screenshots and your own measured results.

Create a README.md in your repo with:

What it does in 3 sentences
The architecture diagram from this page (redraw it in your own style)
data/training_visualisation.png with a sentence on what the red points were
A screenshot of AnomalyDetected in the Alertmanager UI
Your results table from Milestone 6
Limitations, in your own words. For example: cluster-wide averages hide a single bad service, the model has no sense of time of day, and a planned traffic surge looks like an anomaly

A README that states limitations and measured numbers is far more convincing to a hiring manager than one that makes big claims.

Tip
Hiring managers trust measured numbers and stated limitations far more than big claims. Keep your results table and limitations section honest.

Validation & Testing

This final step proves the pipeline works end to end with exact verification commands, then gives debugging steps for the most common failures and a quick reference.

Checklist

Bash

## ─── 1. Data was collected ──────────────────────────────────
ls -lh data/metrics.csv
wc -l data/metrics.csv
## At least 300 lines (about 5 hours at a 60s step)
 
## ─── 2. Model was trained and loads ─────────────────────────
python3 -c "
import joblib
pkg = joblib.load('models/isolation_forest.joblib')
print('Features:', pkg['feature_columns'])
print('Training samples:', pkg['training_samples'])
print('✅ Model loads correctly')
"
 
## ─── 3. A single check works ────────────────────────────────
python3 -c "
from src.collect_metrics import collect_single_point
import joblib
cols = joblib.load('models/isolation_forest.joblib')['feature_columns']
values = collect_single_point(cols)
print('Current metrics:', values)
print('✅ OK' if values else '❌ Cannot collect metrics, check Prometheus')
"
 
## ─── 4. The alert reaches Alertmanager ──────────────────────
## Open http://localhost:9093 after injecting the CPU spike (Milestone 6)
 
## ─── 5. The alert stays firing and then resolves ────────────
## The alert should stay active for the whole anomaly (not vanish after 5 min)
## and disappear soon after recovery.
grep "Back to normal" logs/detection.log
 
## ─── 6. False alerts during normal operation ────────────────
grep -c "ALERT: anomaly confirmed" logs/detection.log
## Record this number in your README
 
echo "✅ Checklist complete"

Debugging Playbook

Too many false positives:

Bash

## Step 1: look at how the training data scores
python3 -c "
import joblib, pandas as pd
pkg = joblib.load('models/isolation_forest.joblib')
df = pd.read_csv('data/metrics.csv')
X = pkg['scaler'].transform(df[pkg['feature_columns']])
scores = pd.Series(pkg['model'].decision_function(X))
print(scores.describe())
print('Share of training rows below 0:', round((scores < 0).mean() * 100, 2), '%')
"
 
## Step 2: lower CONTAMINATION in src/config.py (for example 0.01 -> 0.005),
## or raise CONSECUTIVE_ANOMALIES_REQUIRED, then retrain
python3 -m src.train_model
 
## Step 3: run for another hour and count alerts again
grep -c "ALERT: anomaly confirmed" logs/detection.log

Anomaly injected but nothing detected:

Bash

## Check that the stress is visible in the metric at all
curl -s "http://localhost:9090/api/v1/query?query=avg(rate(container_cpu_usage_seconds_total%7Bcontainer!%3D%22%22%7D%5B5%5D))" | python3 -m json.tool | head -20
 
## Then check the score in the detector log. Scores should trend toward 0
## and then below it. If they barely move, the stress is too small for a
## cluster-wide average: run more stress pods or run them longer.
grep "Score:" logs/detection.log | tail -10

Prometheus connection fails:

Bash

curl -s "http://localhost:9090/api/v1/query?query=up" | python3 -m json.tool | head -5
## Should show "status": "success"
 
## If it fails, start the port-forward:
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 &

Alert posted but nothing in the Alertmanager UI:

Bash

## Check the response directly
curl -i -X POST http://localhost:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{"labels":{"alertname":"ManualTest","severity":"warning"}}]'
## 200 means Alertmanager accepted it. A 404 means you are on the wrong path
## or port-forwarding to the wrong service.

Alert disappears after a few minutes while the anomaly continues:

Bash

## The alert is expiring. Check that the detector is re-sending every cycle:
grep "Alert sent" logs/detection.log | tail -5
## You should see one line per cycle while the anomaly lasts.

Container fails with "Model not found":

Bash

## The model is baked into the image, so train BEFORE building:
ls models/isolation_forest.joblib
docker build -t your-registry/anomaly-detector:v1.0.1 .

Quick Reference

Command	What it does
`python3 -m src.collect_metrics`	Collect training data from Prometheus
`python3 -m src.train_model`	Train the model and save the chart
`python3 -m src.detect_anomalies`	Start continuous detection
`grep "ANOMALOUS" logs/detection.log`	See flagged checks
`grep -c "ALERT: anomaly confirmed" logs/detection.log`	Count confirmed alerts
`kubectl logs -n monitoring deployment/anomaly-detector -f`	Watch the detector in the cluster

Setting (`src/config.py`)	Effect of raising it
`CONTAMINATION`	Looser cut-off, more alerts
`CONSECUTIVE_ANOMALIES_REQUIRED`	Slower but steadier alerts
`TRAINING_DATA_HOURS`	More history, longer collection
`DETECTION_INTERVAL_SECONDS`	Fewer checks, slower detection

Stretch Goals

Persist the model. Store it on a PersistentVolumeClaim and run a Kubernetes CronJob that collects data and retrains weekly.
Add time-of-day features. Add hour_sin and hour_cos columns so the model knows that Monday 9 AM traffic is normal.
Per-service models. Train one model for each important service instead of one for the whole cluster.
Send alerts with labels. Add namespace and service labels so the Incident Response Agent project can diagnose them.
Compare with a threshold alert. Write a normal PromQL threshold alert for the same stress test and compare the time to detect.

Found a mistake, an outdated step or a broken link?

Prometheus Anomaly Detection with Isolation Forest

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem it solves

How the pieces fit

What you will build

What you will learn

What you can show in your portfolio

Before you start

An honest note on limits

What Is Prometheus and Why Are We Using It?

What Is Isolation Forest and Why Are We Using It?

How scoring works (read this once, it matters later)

What Is Alertmanager?

Milestone 1: Set Up the Project and Configuration

Requirements File

Why Configuration Belongs in One File

Milestone 2: Collect Metrics from Prometheus

What This File Does

Milestone 3: Train the Isolation Forest Model

What Is Happening Here

Milestone 4: Send Alerts to Alertmanager

Why a Separate Alert Module

Milestone 5: Run Continuous Detection

What This File Does

Milestone 6: Inject an Anomaly and Measure Detection

Why This Step Matters

Record your results

Milestone 7: Deploy the Detector to Kubernetes

Why Deploy to the Cluster?

Milestone 8: Package It as a Portfolio Project

Validation & Testing

Checklist

Debugging Playbook

Quick Reference

Stretch Goals