KServe Canary Deployment on Kubernetes with kind

Run a KServe canary on kind: split traffic 90/10 between model versions, verify it with real requests, roll back a broken one, promote.

Architecture Overview & Problem Statement

This project runs a KServe canary deployment on a local kind Kubernetes cluster: model v2 receives 10 percent of live traffic while v1 keeps 90 percent, you prove the split with real requests instead of trusting a status field, promote v2 when its error rate and latency hold up, then roll out a broken v3, watch the canary fail, and roll back to v2 with one field. Everything runs free on a laptop; an optional Terraform appendix builds the same thing on Amazon EKS and costs money.

The problem

The acme-fraud team ships a new model by replacing the old one for every user at once. The last release renamed a feature during training (amount became amount_usd) while the payment service still sent amount. Every offline check passed, because offline data had the new column. In production every request failed until someone noticed. A canary would have shown the failure on 10 percent of requests, and a rollback would have taken one field change instead of a hotfix.

What you build

flowchart TD C([Client / traffic_check.py]) --> G[Istio ingress
Host header] G --> R{Knative route
90 / 10} R -->|90%| A[Revision 1
MODEL_VERSION=v1] R -->|10%| B[Revision 2
MODEL_VERSION=v2] A --> D[[canary_decide.py]] B --> D classDef client fill:#DBEAFE,color:#1E293B,stroke:#326CE5 classDef mesh fill:#8B5CF6,color:#FFFFFF,stroke:#8B5CF6 classDef old fill:#475569,color:#FFFFFF,stroke:#475569 classDef new fill:#F97316,color:#FFFFFF,stroke:#F97316 classDef tool fill:#D1FAE5,color:#1E293B,stroke:#10B981 class C client class G,R mesh class A old class B new class D tool

One container image holds all three model versions; an environment variable picks one. Changing it creates a new revision, and canaryTrafficPercent tells KServe how much traffic the newest revision gets. Every response, errors included, carries x-model-version and x-revision headers, so you can count who actually answered.

The rollout you will run

stateDiagram-v2 [*] --> V1: apply isvc-v1 V1 --> Canary_v2: canaryTrafficPercent 10 Canary_v2 --> V2: decide PROMOTE
remove the field V2 --> Canary_v3: v3 at 10% Canary_v3 --> V2_pinned: decide ROLLBACK
canaryTrafficPercent 0

Repository layout

Bash

acme-fraud-kserve/
├── predictor/
│   ├── app.py                 # FastAPI, KServe V1 protocol, version headers
│   ├── Dockerfile
│   ├── requirements.txt
│   └── models/v1 v2 v3/       # written by scripts/train_models.py
├── k8s/
│   ├── kind-config.yaml       # Kubernetes 1.33, pinned by digest
│   ├── isvc-v1.yaml  isvc-v2-canary.yaml  isvc-v2-promote.yaml
│   ├── isvc-v3-canary.yaml  isvc-v3-rollback.yaml  isvc-v9-never-ready.yaml
├── scripts/
│   ├── train_models.py        # v1, v2, and the broken v3
│   ├── cluster_up.sh  build_and_load.sh  ingress.sh  split_status.sh  cluster_down.sh
│   ├── traffic_check.py       # real requests, counted per version
│   └── canary_decide.py       # PROMOTE / ROLLBACK / WAIT
├── tests/test_canary.py
└── terraform/main.tf          # OPTIONAL, costs money

Prerequisites (by module title)

Model Serving on Kubernetes with KServe (custom predictors, V1 protocol, Knative versus Standard mode).
Kubernetes, Docker, and Model Serving with FastAPI and BentoML.
Optional appendix only: Terraform and AWS.
Tools: Docker with at least 4 CPUs and 8 GB of memory for the VM, kind v0.32.0, kubectl 1.33, Python 3.12, curl, sha256sum. The KServe installer fetches Helm and its other helpers itself (read it first, Milestone 1).

Versions used

Component	Version	Checked
KServe (Knative mode)	v0.21.0	GitHub tag and installer, 5 Nov 2026
Istio / Knative Serving / cert-manager	1.27.1 / 1.21.1 / v1.17.0	pinned inside the v0.21.0 installer
kind / Kubernetes node	v0.32.0 / v1.33.12 by digest	kind release notes, 5 Nov 2026
scikit-learn / FastAPI / Python	1.9.1 / 0.162.0 / 3.12	PyPI, 5 Nov 2026

Tip
If you did the KServe module on v0.16, three things changed by v0.21. "Serverless" mode is now called Knative mode (the old value still works but is deprecated). The default mode is now Standard, which does not support canaryTrafficPercent, so every manifest here sets serving.kserve.io/deploymentMode: Knative explicitly. And the old hack/quick_install.sh was replaced by scripts under hack/setup/quick-install/.

Tip
What was run for this guide: model training, the predictor (readiness, prediction, a 422 for bad input, the v3 feature-mismatch 500), traffic_check.py and canary_decide.py against a sandbox stand-in for Knative's weighted router (90/10, 100/0, a broken 10 percent canary, and a deliberately wrong 70/30 split), and the 4 tests. The kind cluster, the KServe install, the image build and every kubectl step were not run: the sandbox cannot pull container images. Those steps show the expected output shape, from the KServe and Knative documentation, and are marked "not run". The Terraform appendix was not run or validated.

Milestone 1: Create the kind Cluster and Install KServe

This milestone creates a one-node Kubernetes 1.33 cluster with kind and installs KServe v0.21.0 in Knative mode with its pinned dependencies, after you have read the installer.

Why Knative mode for a canary

KServe has two ways to run a model. Standard mode creates a plain Deployment: simple, but one version at a time. Knative mode runs the predictor as a Knative Service, where every spec change becomes an immutable revision and the route can split traffic between revisions by percentage. canaryTrafficPercent is a Knative-mode feature.

Step 1: Write the kind config

k8s/kind-config.yaml:

YAML

# One-node kind cluster for KServe in Knative mode. Give Docker at least 4 CPUs and 8 GB.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: acme-kserve
nodes:
  - role: control-plane
    # Kubernetes 1.33, pinned by digest (kind v0.32.0 release notes)
    image: kindest/node:v1.33.12@sha256:3f5c8443c620245e4d355cfe09e96a91ead32ceaa569d3f1ca9edf0cb2fe2ff4

Step 2: Create the cluster and install KServe

scripts/cluster_up.sh:

Bash

#!/usr/bin/env bash
# Create the kind cluster and install KServe v0.21.0 in Knative mode.
# Installs: Gateway API CRDs, cert-manager v1.17.0, Istio 1.27.1, Knative Serving 1.21.1, KServe v0.21.0.
set -euo pipefail
KSERVE_VERSION=v0.21.0
SCRIPT=kserve-knative-mode-full-install-helm.sh
URL="https://raw.githubusercontent.com/kserve/kserve/${KSERVE_VERSION}/hack/setup/quick-install/${SCRIPT}"
 
kind create cluster --config k8s/kind-config.yaml
kubectl cluster-info --context kind-acme-kserve
 
# Download the installer pinned to the release tag, keep a copy, and READ IT before running.
mkdir -p .install
curl -fsSL "$URL" -o ".install/${SCRIPT}"
sha256sum ".install/${SCRIPT}" | tee .install/installer.sha256
if [ "${I_HAVE_READ_THE_INSTALLER:-no}" != "yes" ]; then
  echo "Review .install/${SCRIPT}, then rerun with I_HAVE_READ_THE_INSTALLER=yes"; exit 3
fi
bash ".install/${SCRIPT}"
 
kubectl wait --for=condition=Available deployment --all -n kserve --timeout=600s
kubectl create namespace acme-fraud --dry-run=client -o yaml | kubectl apply -f -
echo "cluster ready"

Bash

./scripts/cluster_up.sh                         # stops after downloading: read the installer
less .install/kserve-knative-mode-full-install-helm.sh
I_HAVE_READ_THE_INSTALLER=yes ./scripts/cluster_up.sh

Security
Never pipe a remote script straight into bash. The script is pinned to the v0.21.0 tag, saved to disk, hashed, and only run after you read it. Keep .install/installer.sha256 so you can tell later if a rerun fetched something different.

Step 3: Check the platform

Bash

kubectl get pods -n kserve
kubectl get pods -n knative-serving
kubectl get pods -n istio-system

Expected output shape (not run for this guide): every pod Running and READY 1/1 (or 2/2), including kserve-controller-manager, activator, autoscaler, controller, webhook and istio-ingressgateway.

Acceptance check

kubectl get crd inferenceservices.serving.kserve.io exists.
All pods in kserve, knative-serving and istio-system are ready.

Common Mistake
Giving Docker Desktop 2 CPUs and 4 GB. Istio, Knative and KServe together leave pods Pending with Insufficient cpu. Raise the VM to at least 4 CPUs and 8 GB before creating the cluster.

Milestone 2: Build the Predictor and Load It into kind

This milestone trains three model versions, packages them into one predictor image that speaks KServe's V1 protocol, and loads the image into kind without a registry.

Why one image with three versions

Here the image is fixed and only MODEL_VERSION changes, so every revision differs in exactly one thing: the model. That keeps the experiment clean. In production you would pull the model from a registry or storage URI instead, but the canary mechanics are identical.

Step 1: Train the versions

scripts/train_models.py:

Python

"""Train the three model versions this project rolls out.
 
v1: the current champion (max_leaf_nodes=31)
v2: the candidate (max_leaf_nodes=7, picked on validation)
v3: a BROKEN candidate: trained after a feature was renamed (amount -> amount_usd),
    while clients still send "amount". It passes offline checks and fails live.
"""
import hashlib
import json
from pathlib import Path
 
import joblib
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
 
CATEGORIES = ["grocery", "electronics", "travel", "fuel", "fashion", "gaming"]
 
 
def make_payments(rows=20000, fraud_rate=0.03, seed=42) -> pd.DataFrame:
    """Same synthetic payments as the earlier projects."""
    rng = np.random.default_rng(seed)
    df = pd.DataFrame({
        "amount": np.round(rng.lognormal(3.5, 1.0, rows), 2),
        "hour": rng.integers(0, 24, rows),
        "merchant_category": rng.choice(CATEGORIES, rows),
        "country_mismatch": rng.binomial(1, 0.08, rows),
        "card_age_days": rng.integers(1, 3650, rows),
        "txns_last_24h": rng.poisson(2.0, rows),
    })
    score = (0.9 * np.log1p(df["amount"]) / 5 + 1.2 * df["country_mismatch"]
             + 0.6 * df["merchant_category"].isin(["electronics", "gaming"])
             + 0.5 * ((df["hour"] <= 4) | (df["hour"] >= 23))
             + 0.25 * np.minimum(df["txns_last_24h"], 8) / 2
             - 0.6 * (df["card_age_days"] > 365) + rng.normal(0, 1.0, rows))
    df["is_fraud"] = (score >= np.quantile(score, 1 - fraud_rate)).astype(int)
    return df
 
 
def train(df: pd.DataFrame, max_leaf_nodes: int):
    features = [c for c in df.columns if c != "is_fraud"]
    tr, va = train_test_split(df, test_size=0.2, stratify=df["is_fraud"], random_state=42)
    model = Pipeline([
        ("pre", ColumnTransformer([("cat", OneHotEncoder(handle_unknown="ignore"), ["merchant_category"])],
                                  remainder="passthrough")),
        ("clf", HistGradientBoostingClassifier(max_leaf_nodes=max_leaf_nodes, class_weight="balanced",
                                               random_state=42)),
    ]).fit(tr[features], tr["is_fraud"])
    return model, average_precision_score(va["is_fraud"], model.predict_proba(va[features])[:, 1])
 
 
def save(version: str, model, pr_auc: float) -> None:
    out = Path("predictor/models") / version
    out.mkdir(parents=True, exist_ok=True)
    joblib.dump(model, out / "model.joblib")
    meta = {"version": version, "val_pr_auc": round(pr_auc, 4), "threshold": 0.5,
            "model_sha256": hashlib.sha256((out / "model.joblib").read_bytes()).hexdigest()}
    (out / "metadata.json").write_text(json.dumps(meta, indent=2))
    print(f"{version}: val_pr_auc={pr_auc:.4f} sha={meta['model_sha256'][:12]}")
 
 
if __name__ == "__main__":
    df = make_payments()
    save("v1", *train(df, 31))
    save("v2", *train(df, 7))
    save("v3", *train(df.rename(columns={"amount": "amount_usd"}), 7))   # the bug

Bash

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r predictor/requirements.txt pytest
python scripts/train_models.py

Expected output (real run):

TEXT

v1: val_pr_auc=0.2159 sha=0759b53f1f5b
v2: val_pr_auc=0.2335 sha=81412ae5fa05
v3: val_pr_auc=0.2335 sha=276fe81ba5dd

v2's smaller trees were picked on validation data. v3 is the same model trained after the column rename: identical offline score, broken online. That is the whole point of a canary.

Step 2: Write the predictor

predictor/app.py:

Python

"""acme-fraud predictor for KServe: a custom container speaking the V1 protocol.
 
GET  /v1/models/acme-fraud           -> 200 when the model is loaded (readiness)
POST /v1/models/acme-fraud:predict   -> {"predictions": [...], "model_version", "revision"}
 
Which model it serves comes from the MODEL_VERSION env var. Knative sets K_REVISION,
which we echo back so every response says exactly which revision answered.
"""
import hashlib
import json
import os
from contextlib import asynccontextmanager
from pathlib import Path
from typing import Literal
 
import joblib
import pandas as pd
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
 
MODEL_NAME = "acme-fraud"
VERSION = os.environ.get("MODEL_VERSION", "v1")
REVISION = os.environ.get("K_REVISION", "local")      # set by Knative inside the cluster
MODEL_DIR = Path(os.environ.get("MODEL_ROOT", "/app/models")) / VERSION
STATE: dict = {}
 
 
class Payment(BaseModel):
    amount: float = Field(gt=0, le=1_000_000)
    hour: int = Field(ge=0, le=23)
    merchant_category: Literal["grocery", "electronics", "travel", "fuel", "fashion", "gaming"]
    country_mismatch: Literal[0, 1]
    card_age_days: int = Field(ge=0, le=36_500)
    txns_last_24h: int = Field(ge=0, le=1_000)
 
 
class PredictRequest(BaseModel):          # V1 protocol: {"instances": [...]}
    instances: list[Payment] = Field(min_length=1, max_length=100)
 
 
@asynccontextmanager
async def lifespan(app: FastAPI):
    meta = json.loads((MODEL_DIR / "metadata.json").read_text())
    blob = (MODEL_DIR / "model.joblib").read_bytes()
    if hashlib.sha256(blob).hexdigest() != meta["model_sha256"]:
        raise RuntimeError("model file does not match its metadata")   # pod never becomes ready
    STATE["model"], STATE["meta"] = joblib.load(MODEL_DIR / "model.joblib"), meta
    yield
 
 
app = FastAPI(title="acme-fraud-predictor", lifespan=lifespan)
 
 
@app.middleware("http")
async def tag_response(request, call_next):
    """Stamp EVERY response, errors included, with the version and revision that produced it."""
    response = await call_next(request)
    response.headers["x-model-version"] = VERSION
    response.headers["x-revision"] = REVISION
    return response
 
 
@app.get(f"/v1/models/{MODEL_NAME}")
def model_ready() -> dict:
    return {"name": MODEL_NAME, "ready": "model" in STATE, "model_version": VERSION}
 
 
@app.post(f"/v1/models/{MODEL_NAME}:predict")
def predict(req: PredictRequest) -> dict:
    rows = pd.DataFrame([p.model_dump() for p in req.instances])
    model = STATE["model"]
    try:
        # select columns by the names the model was TRAINED with
        proba = model.predict_proba(rows[list(model.feature_names_in_)])[:, 1]
    except KeyError as err:
        # training-serving skew: the model expects a feature the request does not have
        raise HTTPException(status_code=500, detail=f"feature mismatch: {err}") from err
    t = STATE["meta"]["threshold"]
    return {
        "predictions": [{"fraud_probability": round(float(p), 4), "is_fraud": bool(p >= t)} for p in proba],
        "model_version": VERSION,
        "revision": REVISION,
    }

predictor/requirements.txt:

TEXT

fastapi==0.162.0
uvicorn==0.54.0
pydantic==2.14.2
scikit-learn==1.9.1
pandas==3.1.1
numpy==2.5.4
joblib==1.6.0

predictor/Dockerfile:

Dockerfile

FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 OMP_NUM_THREADS=1 MODEL_ROOT=/app/models
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
# all three versions are baked in; MODEL_VERSION picks one at runtime
COPY models/ models/
RUN useradd --create-home --uid 10001 app
USER 10001
EXPOSE 8080
# KServe sends traffic to port 8080 by default for custom containers
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "1"]

Step 3: Smoke test it locally before Kubernetes

Bash

cd predictor
MODEL_ROOT=$PWD/models MODEL_VERSION=v1 K_REVISION=acme-fraud-predictor-00001 uvicorn app:app --port 8081 &
MODEL_ROOT=$PWD/models MODEL_VERSION=v3 K_REVISION=acme-fraud-predictor-00003 uvicorn app:app --port 8083 &
cd ..
BODY='{"instances":[{"amount":129.99,"hour":2,"merchant_category":"electronics","country_mismatch":1,"card_age_days":40,"txns_last_24h":6}]}'
curl -s localhost:8081/v1/models/acme-fraud; echo
curl -s -i -X POST localhost:8081/v1/models/acme-fraud:predict -H 'Content-Type: application/json' -d "$BODY" | grep -iE "^x-|predictions"
curl -s -i -X POST localhost:8083/v1/models/acme-fraud:predict -H 'Content-Type: application/json' -d "$BODY" | grep -iE "^HTTP|^x-|detail"
kill %1 %2

Expected output (real run; revision names were set by hand here, Knative sets them in the cluster):

JSON

{"name":"acme-fraud","ready":true,"model_version":"v1"}
x-model-version: v1
x-revision: acme-fraud-predictor-00001
{"predictions":[{"fraud_probability":0.861,"is_fraud":true}],"model_version":"v1","revision":"acme-fraud-predictor-00001"}
HTTP/1.1 500 Internal Server Error
x-model-version: v3
x-revision: acme-fraud-predictor-00003
{"detail":"feature mismatch: \"['amount_usd'] not in index\""}

v3 is ready, healthy by every probe, and fails every prediction. Readiness probes cannot catch this; only real traffic can.

Step 4: Build and load into kind

scripts/build_and_load.sh:

Bash

#!/usr/bin/env bash
# Train the three versions, build one image holding them, and load it into kind (no registry needed).
set -euo pipefail
IMAGE=dev.local/acme-fraud-predictor:0.1.0      # dev.local: Knative will not try to resolve a digest
python scripts/train_models.py
docker build -t "$IMAGE" predictor/
kind load docker-image "$IMAGE" --name acme-kserve
echo "loaded $IMAGE"

Bash

./scripts/build_and_load.sh
docker exec acme-kserve-control-plane crictl images | grep acme-fraud

Expected output shape (not run): one line showing dev.local/acme-fraud-predictor with tag 0.1.0.

Common Mistake
Naming the image acme-fraud-predictor:0.1.0 without a registry prefix. Knative resolves every image tag to a digest by asking the registry, and a kind-loaded image is in no registry, so the revision fails with "failed to resolve image to digest". Knative skips that lookup for kind.local, ko.local and dev.local (the registries-skipping-tag-resolving default in Knative 1.21), which is why the image is called dev.local/....

Milestone 3: Deploy v1 and Verify It Two Ways

This milestone deploys v1 as an InferenceService and checks the traffic in two independent ways: what KServe reports, and what real requests through the ingress actually hit.

Why two checks

READY=True and the traffic field describe what the control plane intends. They do not prove that the ingress, the Host header and the routes deliver requests where you think. traffic_check.py sends real requests and counts the x-model-version header on each answer.

Step 1: Write the manifests

k8s/isvc-v1.yaml:

YAML

# Step 1: v1 takes 100% of traffic.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: acme-fraud
  namespace: acme-fraud
  annotations:
    serving.kserve.io/deploymentMode: Knative         # canary needs Knative mode; v0.21 default is Standard
    serving.kserve.io/enable-tag-routing: "true"      # adds latest- and prev- hostnames for debugging
spec:
  predictor:
    minReplicas: 1                                    # keep one pod warm, so the canary is not a cold start
    maxReplicas: 2
    containers:
      - name: kserve-container
        image: dev.local/acme-fraud-predictor:0.1.0   # dev.local: Knative skips digest lookup for kind-loaded images
        imagePullPolicy: IfNotPresent
        env:
          - name: MODEL_VERSION
            value: "v1"
        ports:
          - containerPort: 8080
            protocol: TCP
        readinessProbe:
          httpGet:
            path: /v1/models/acme-fraud
        resources:
          requests: {cpu: 100m, memory: 256Mi}
          limits: {cpu: "1", memory: 512Mi}

The other five manifests differ only in MODEL_VERSION and the canaryTrafficPercent line (shown in later milestones).

Step 2: Write the verification tools

scripts/split_status.sh:

Bash

#!/usr/bin/env bash
# What KServe says the split is (check 1 of 2; traffic_check.py is check 2).
set -euo pipefail
kubectl get isvc acme-fraud -n acme-fraud \
  -o jsonpath='{range .status.components.predictor.traffic[*]}{.revisionName}{"  "}{.percent}{"%  tag="}{.tag}{"\n"}{end}'
kubectl get isvc acme-fraud -n acme-fraud \
  -o jsonpath='latestReady={.status.components.predictor.latestReadyRevision}  latestRolledOut={.status.components.predictor.latestRolledoutRevision}  previousRolledOut={.status.components.predictor.previousRolledoutRevision}{"\n"}'

scripts/ingress.sh:

Bash

#!/usr/bin/env bash
# Open the Istio ingress on localhost:8080 and print the Host header to use.
# Usage: source scripts/ingress.sh   (exports ISVC_HOST and PF_PID)
kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80 >/dev/null 2>&1 &
export PF_PID=$!
sleep 3
ISVC_HOST=$(kubectl get isvc acme-fraud -n acme-fraud -o jsonpath='{.status.url}' | sed 's|https\?://||')
export ISVC_HOST
echo "ingress on localhost:8080 (pid ${PF_PID}), Host: ${ISVC_HOST}"

scripts/traffic_check.py:

Python

"""Send real requests through the ingress and count who answered.
 
Do not trust READY or the traffic field alone: this measures the split users actually get.
Exit codes: 0 = observed canary share within the expected band, 1 = outside it, 4 = bad input.
"""
import argparse
import json
import math
import random
import sys
import time
import urllib.error
import urllib.request
from collections import defaultdict
from pathlib import Path
 
CATS = ["grocery", "electronics", "travel", "fuel", "fashion", "gaming"]
 
 
def one_request(url: str, host: str | None) -> tuple[str, int, float]:
    body = json.dumps({"instances": [{
        "amount": round(random.lognormvariate(3.5, 1.0), 2), "hour": random.randint(0, 23),
        "merchant_category": random.choice(CATS), "country_mismatch": random.choice([0, 0, 0, 1]),
        "card_age_days": random.randint(1, 3650), "txns_last_24h": random.randint(0, 10)}]}).encode()
    req = urllib.request.Request(url, data=body, headers={"Content-Type": "application/json"})
    if host:
        req.add_header("Host", host)              # Istio routes on the Host header
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=10) as r:
            status, version = r.status, r.headers.get("x-model-version", "?")
    except urllib.error.HTTPError as err:          # 4xx/5xx still carry our headers
        status, version = err.code, err.headers.get("x-model-version", "?")
    except (urllib.error.URLError, TimeoutError):
        status, version = 0, "no-response"
    return version, status, (time.perf_counter() - start) * 1000
 
 
def p95(values: list[float]) -> float:
    s = sorted(values)
    return round(s[max(0, math.ceil(0.95 * len(s)) - 1)], 1) if s else 0.0
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--url", default="http://localhost:8080/v1/models/acme-fraud:predict")
    ap.add_argument("--host", help="Host header, e.g. acme-fraud.acme-fraud.example.com")
    ap.add_argument("--requests", type=int, default=500)
    ap.add_argument("--canary-version", required=True)
    ap.add_argument("--expect-canary-percent", type=float, required=True)
    ap.add_argument("--out", required=True)
    a = ap.parse_args()
 
    stats = defaultdict(lambda: {"count": 0, "errors": 0, "lat": []})
    for _ in range(a.requests):
        version, status, ms = one_request(a.url, a.host)
        s = stats[version]
        s["count"] += 1
        s["errors"] += status != 200
        s["lat"].append(ms)
 
    n = a.requests
    report = {v: {"count": s["count"], "share": round(s["count"] / n, 4), "errors": s["errors"],
                  "error_rate": round(s["errors"] / s["count"], 4), "p95_ms": p95(s["lat"])}
              for v, s in sorted(stats.items())}
    for v, r in report.items():
        print(f"{v:<12} {r['count']:>5} reqs  share {100*r['share']:5.1f}%  "
              f"errors {r['errors']:>4} ({100*r['error_rate']:.1f}%)  p95 {r['p95_ms']} ms")
 
    # 3-sigma band for a binomial share: wide enough not to flap, tight enough to catch a wrong split
    p = a.expect_canary_percent / 100
    sigma = math.sqrt(p * (1 - p) / n) if 0 < p < 1 else 0.0
    lo, hi = max(0.0, p - 3 * sigma), min(1.0, p + 3 * sigma)
    seen = report.get(a.canary_version, {"share": 0.0})["share"]
    ok = lo <= seen <= hi
    print(f"canary {a.canary_version}: observed {100*seen:.1f}%, expected {100*p:.0f}% "
          f"(band {100*lo:.1f}% to {100*hi:.1f}%) -> {'SPLIT OK' if ok else 'SPLIT WRONG'}")
    Path(a.out).parent.mkdir(parents=True, exist_ok=True)
    Path(a.out).write_text(json.dumps({"requests": n, "expected_canary": p, "canary_version": a.canary_version,
                                       "split_ok": ok, "by_version": report}, indent=2))
    return 0 if ok else 1
 
 
if __name__ == "__main__":
    sys.exit(main())

The pass band is three standard deviations of a binomial share: for 500 requests and an expected 10 percent, anything from 6.0 to 14.0 percent is consistent with a true 10 percent split. Narrower bands flap; wider ones miss real misrouting.

Step 3: Deploy and check

Bash

kubectl apply -f k8s/isvc-v1.yaml
kubectl wait --for=condition=Ready isvc/acme-fraud -n acme-fraud --timeout=300s
./scripts/split_status.sh
source scripts/ingress.sh
python scripts/traffic_check.py --host "$ISVC_HOST" --canary-version v2 --expect-canary-percent 0 --out results/v1-only.json

Expected output shape (not run): split_status.sh prints one revision, acme-fraud-predictor-00001 100% tag=, and traffic_check.py prints v1 500 reqs share 100.0% errors 0, then SPLIT OK.

Acceptance check

kubectl get isvc -n acme-fraud shows READY True and a URL.
500 out of 500 responses carry x-model-version: v1.

Milestone 4: Canary v2 at 10 Percent and Decide

This milestone sends 10 percent of traffic to v2, verifies the split with real requests, and lets a rule written in advance decide whether to promote.

Step 1: Write the decision rules

scripts/canary_decide.py:

Python

"""Decide promote, rollback or wait from a traffic_check report.
 
Rules were written before any canary ran. Illustrative, not production SLOs:
  * need at least 30 canary responses, otherwise WAIT
  * canary error rate <= baseline error rate + 1 percentage point
  * canary p95 latency <= 1.5 x baseline p95 + 20 ms
Exit codes: 0 = PROMOTE, 1 = ROLLBACK, 2 = WAIT (not enough data), 4 = bad input.
"""
import argparse
import json
import sys
from pathlib import Path
 
MIN_SAMPLES, MAX_EXTRA_ERRORS, P95_FACTOR, P95_SLACK_MS = 30, 0.01, 1.5, 20
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--report", required=True)
    ap.add_argument("--baseline-version", required=True)
    ap.add_argument("--canary-version", required=True)
    a = ap.parse_args()
    r = json.loads(Path(a.report).read_text())["by_version"]
    if a.baseline_version not in r:
        print(f"ERROR: no responses from baseline {a.baseline_version}"); return 4
    base, can = r[a.baseline_version], r.get(a.canary_version)
    if can is None or can["count"] < MIN_SAMPLES:
        print(f"WAIT: {0 if can is None else can['count']} canary responses, need {MIN_SAMPLES}"); return 2
 
    err_ok = can["error_rate"] <= base["error_rate"] + MAX_EXTRA_ERRORS
    lat_limit = P95_FACTOR * base["p95_ms"] + P95_SLACK_MS
    lat_ok = can["p95_ms"] <= lat_limit
    print(f"{'PASS' if err_ok else 'FAIL'}  error rate: canary {100*can['error_rate']:.1f}% "
          f"vs limit {100*(base['error_rate'] + MAX_EXTRA_ERRORS):.1f}%")
    print(f"{'PASS' if lat_ok else 'FAIL'}  p95: canary {can['p95_ms']} ms vs limit {lat_limit:.1f} ms")
    if err_ok and lat_ok:
        print("DECISION: PROMOTE"); return 0
    print("DECISION: ROLLBACK"); return 1
 
 
if __name__ == "__main__":
    sys.exit(main())

tests/test_canary.py:

Python

"""Decision rules and the split band, tested without a cluster."""
import json
import subprocess
import sys
 
 
def decide(tmp_path, base: dict, canary: dict | None) -> int:
    by = {"v1": {"count": 450, **base}}
    if canary:
        by["v2"] = canary
    (tmp_path / "r.json").write_text(json.dumps({"by_version": by}))
    return subprocess.run([sys.executable, "scripts/canary_decide.py", "--report", str(tmp_path / "r.json"),
                           "--baseline-version", "v1", "--canary-version", "v2"], capture_output=True).returncode
 
 
OK = {"error_rate": 0.0, "p95_ms": 10.0}
 
 
def test_healthy_canary_is_promoted(tmp_path):
    assert decide(tmp_path, OK, {"count": 50, "error_rate": 0.0, "p95_ms": 12.0}) == 0
 
 
def test_erroring_canary_is_rolled_back(tmp_path):
    assert decide(tmp_path, OK, {"count": 50, "error_rate": 0.04, "p95_ms": 9.0}) == 1
 
 
def test_slow_canary_is_rolled_back(tmp_path):
    assert decide(tmp_path, OK, {"count": 50, "error_rate": 0.0, "p95_ms": 60.0}) == 1
 
 
def test_too_few_canary_responses_means_wait(tmp_path):
    assert decide(tmp_path, OK, {"count": 12, "error_rate": 0.0, "p95_ms": 9.0}) == 2
    assert decide(tmp_path, OK, None) == 2

Bash

pytest -q tests

Expected output (real run):

TEXT

....                                                                     [100%]
4 passed in 0.19s

Step 2: Start the canary

k8s/isvc-v2-canary.yaml:

YAML

# Step 2: canary v2 at 10%.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: acme-fraud
  namespace: acme-fraud
  annotations:
    serving.kserve.io/deploymentMode: Knative         # canary needs Knative mode; v0.21 default is Standard
    serving.kserve.io/enable-tag-routing: "true"      # adds latest- and prev- hostnames for debugging
spec:
  predictor:
    minReplicas: 1                                    # keep one pod warm, so the canary is not a cold start
    maxReplicas: 2
    canaryTrafficPercent: 10                          # new revision gets 10%, last rolled-out keeps 90%
    containers:
      - name: kserve-container
        image: dev.local/acme-fraud-predictor:0.1.0   # dev.local: Knative skips digest lookup for kind-loaded images
        imagePullPolicy: IfNotPresent
        env:
          - name: MODEL_VERSION
            value: "v2"
        ports:
          - containerPort: 8080
            protocol: TCP
        readinessProbe:
          httpGet:
            path: /v1/models/acme-fraud
        resources:
          requests: {cpu: 100m, memory: 256Mi}
          limits: {cpu: "1", memory: 512Mi}

Bash

kubectl apply -f k8s/isvc-v2-canary.yaml
kubectl wait --for=condition=Ready isvc/acme-fraud -n acme-fraud --timeout=300s
./scripts/split_status.sh

Expected output shape (not run; from the KServe rollout docs):

TEXT

acme-fraud-predictor-00002  10%  tag=latest
acme-fraud-predictor-00001  90%  tag=prev
latestReady=acme-fraud-predictor-00002  latestRolledOut=acme-fraud-predictor-00001  previousRolledOut=acme-fraud-predictor-00001

Step 3: Verify with traffic, then decide

Bash

python scripts/traffic_check.py --host "$ISVC_HOST" --canary-version v2 --expect-canary-percent 10 --out results/canary-v2.json
python scripts/canary_decide.py --report results/canary-v2.json --baseline-version v1 --canary-version v2

Expected output (real run against the sandbox's stand-in router, not Knative; your counts will vary around 10 percent):

◈ DIAGRAM

v1             451 reqs  share  90.2%  errors    0 (0.0%)  p95 8.3 ms
v2              49 reqs  share   9.8%  errors    0 (0.0%)  p95 10.0 ms
canary v2: observed 9.8%, expected 10% (band 6.0% to 14.0%) -> SPLIT OK
PASS  error rate: canary 0.0% vs limit 1.0%
PASS  p95: canary 10.0 ms vs limit 32.5 ms
DECISION: PROMOTE

49 canary responses is enough for this rule (minimum 30) but thin evidence for rare errors: one error in 49 is already 2 percent. Real canaries run longer, or at a higher percentage once the first minutes look clean.

Tip
With tag routing on, you can hit each revision directly to debug it: curl -H "Host: latest-$ISVC_HOST" ... for the canary and prev-$ISVC_HOST for the stable revision. Exact tag hostnames are in kubectl get isvc acme-fraud -n acme-fraud -o jsonpath='{.status.components.predictor.traffic}'.

Acceptance check

Both checks agree: the status says 10/90, and the observed canary share is inside the band.
canary_decide.py printed PROMOTE and exited 0.

Milestone 5: Promote v2

This milestone promotes v2 by removing canaryTrafficPercent, which sends 100 percent to the latest ready revision, and proves no request still reaches v1.

Step 1: Promote

k8s/isvc-v2-promote.yaml is isvc-v2-canary.yaml without the canaryTrafficPercent line.

Bash

kubectl apply -f k8s/isvc-v2-promote.yaml
kubectl wait --for=condition=Ready isvc/acme-fraud -n acme-fraud --timeout=300s
./scripts/split_status.sh
python scripts/traffic_check.py --host "$ISVC_HOST" --canary-version v1 --expect-canary-percent 0 --out results/promoted-v2.json

Expected output (traffic part real, against the stand-in router):

◈ DIAGRAM

v2             500 reqs  share 100.0%  errors    0 (0.0%)  p95 6.3 ms
canary v1: observed 0.0%, expected 0% (band 0.0% to 0.0%) -> SPLIT OK

Status shape (not run): one revision, acme-fraud-predictor-00002 100%, and latestRolledOut=acme-fraud-predictor-00002. KServe now treats revision 2 as the last good rollout, which is what a later rollback returns to.

Acceptance check

latestRolledoutRevision is revision 2.
0 of 500 responses came from v1.

Milestone 6: Canary the Broken v3 and Roll Back

This milestone rolls out v3 at 10 percent, lets the error rate condemn it, and rolls back by setting canaryTrafficPercent: 0, which pins all traffic to the last rolled-out revision.

Step 1: Start the v3 canary and measure

Bash

kubectl apply -f k8s/isvc-v3-canary.yaml
kubectl wait --for=condition=Ready isvc/acme-fraud -n acme-fraud --timeout=300s
python scripts/traffic_check.py --host "$ISVC_HOST" --canary-version v3 --expect-canary-percent 10 --out results/canary-v3.json
python scripts/canary_decide.py --report results/canary-v3.json --baseline-version v2 --canary-version v3; echo "exit=$?"

Expected output (real run, stand-in router):

◈ DIAGRAM

v2             448 reqs  share  89.6%  errors    0 (0.0%)  p95 6.5 ms
v3              52 reqs  share  10.4%  errors   52 (100.0%)  p95 5.2 ms
canary v3: observed 10.4%, expected 10% (band 6.0% to 14.0%) -> SPLIT OK
FAIL  error rate: canary 100.0% vs limit 1.0%
PASS  p95: canary 5.2 ms vs limit 29.8 ms
DECISION: ROLLBACK
exit=1

Notice the latency check passed: failing fast is fast. A canary judged on latency alone would have promoted this model.

Step 2: Roll back

k8s/isvc-v3-rollback.yaml:

YAML

# Step 5: roll back. Same spec as v3, canary set to 0%.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: acme-fraud
  namespace: acme-fraud
  annotations:
    serving.kserve.io/deploymentMode: Knative         # canary needs Knative mode; v0.21 default is Standard
    serving.kserve.io/enable-tag-routing: "true"      # adds latest- and prev- hostnames for debugging
spec:
  predictor:
    minReplicas: 1                                    # keep one pod warm, so the canary is not a cold start
    maxReplicas: 2
    canaryTrafficPercent: 0                           # pin: 100% back to the last rolled-out revision (v2)
    containers:
      - name: kserve-container
        image: dev.local/acme-fraud-predictor:0.1.0   # dev.local: Knative skips digest lookup for kind-loaded images
        imagePullPolicy: IfNotPresent
        env:
          - name: MODEL_VERSION
            value: "v3"
        ports:
          - containerPort: 8080
            protocol: TCP
        readinessProbe:
          httpGet:
            path: /v1/models/acme-fraud
        resources:
          requests: {cpu: 100m, memory: 256Mi}
          limits: {cpu: "1", memory: 512Mi}

Bash

kubectl apply -f k8s/isvc-v3-rollback.yaml
./scripts/split_status.sh
python scripts/traffic_check.py --host "$ISVC_HOST" --canary-version v3 --expect-canary-percent 0 --out results/rolled-back.json

Expected output (traffic part real, stand-in router):

◈ DIAGRAM

v2             500 reqs  share 100.0%  errors    0 (0.0%)  p95 7.0 ms
canary v3: observed 0.0%, expected 0% (band 0.0% to 0.0%) -> SPLIT OK

Status shape (not run): revision 3 at 0% with tag=latest, revision 2 at 100% with tag=prev.

Step 3: Leave a clean state

The rollback pins traffic but leaves the broken spec as the latest. Re-apply the good spec so the next change starts from v2:

Bash

kubectl apply -f k8s/isvc-v2-promote.yaml
kubectl wait --for=condition=Ready isvc/acme-fraud -n acme-fraud --timeout=300s
./scripts/split_status.sh

Knative creates a new revision (4) for this spec change even though it matches revision 2, and sends it 100 percent.

Acceptance check

canary_decide.py exited 1 for v3, and the reason was the error rate.
After rollback, 0 of 500 responses came from v3.

Common Mistake
Rolling back by deleting the InferenceService and re-creating it with v2. That drops all traffic during the re-create and throws away revision history. Setting canaryTrafficPercent: 0 moves traffic in one route update while both revisions stay up.

Break It on Purpose

The broken model in Milestone 6 is the main exercise. These two break the platform instead.

Step 1: A revision that never becomes ready

k8s/isvc-v9-never-ready.yaml asks for MODEL_VERSION=v9, which does not exist in the image, at 10 percent:

Bash

kubectl apply -f k8s/isvc-v9-never-ready.yaml
kubectl get isvc acme-fraud -n acme-fraud
kubectl get pods -n acme-fraud
python scripts/traffic_check.py --host "$ISVC_HOST" --canary-version v9 --expect-canary-percent 0 --out results/v9.json

Expected behaviour (not run): the new pod crashes on startup (FileNotFoundError for /app/models/v9/metadata.json, status CrashLoopBackOff), the InferenceService shows READY False, and traffic stays 100 percent on the previous revision. Knative never routes to a revision that is not ready, so a crash-on-start bug cannot reach users; a bug like v3's, which starts fine and fails on requests, can. After Knative's progress-deadline (600 seconds by default) the revision is marked failed. Re-apply k8s/isvc-v2-promote.yaml to recover.

Step 2: An image Knative cannot resolve

Change the image in a copy of isvc-v2-promote.yaml to acme-fraud-predictor:0.1.0 (no dev.local/), tag it the same way locally, load it into kind, and apply it.

Expected behaviour (not run): the revision fails with a message about resolving the image to a digest, even though the image is on the node. Put dev.local/ back.

Step 3: Clean up after breaking

Bash

kubectl apply -f k8s/isvc-v2-promote.yaml
kubectl wait --for=condition=Ready isvc/acme-fraud -n acme-fraud --timeout=300s
./scripts/split_status.sh                       # one revision at 100%

Validation & Testing

Run these to prove the project end to end, then tear down. The local route costs nothing.

Step 1: Final checks

Bash

pytest -q tests                                                   # 4 passed
kubectl get revisions -n acme-fraud                               # history of every rollout step
ls results/                                                       # one JSON per traffic check
python scripts/traffic_check.py --host "$ISVC_HOST" --canary-version v2 --expect-canary-percent 10 \
  --out results/sanity.json; echo "exit=$?"                       # on a 100/0 route this must FAIL

The last command is a sanity check of the checker itself: against a route that is 100 percent on one version it must print SPLIT WRONG and exit 1. In the sandbox, a deliberately wrong 70/30 split printed (real run):

◈ DIAGRAM

canary v2: observed 28.0%, expected 10% (band 6.0% to 14.0%) -> SPLIT WRONG

Definition of done

KServe v0.21.0 runs in Knative mode on kind, installed from a pinned installer you read.
Every manifest sets deploymentMode: Knative and uses a dev.local/ image.
A 90/10 split was verified by status and by real requests, inside the 3-sigma band.
v2 was promoted by rule (PROMOTE), v3 was rolled back by rule (ROLLBACK), and both steps were re-verified with traffic.
The README shows the results/*.json files and states what ran on a real cluster.

Step 2: Tear down

scripts/cluster_down.sh:

Bash

#!/usr/bin/env bash
# Delete the kind cluster and the local image. Nothing else was created.
set -euo pipefail
[ -n "${PF_PID:-}" ] && kill "$PF_PID" 2>/dev/null || true
kind delete cluster --name acme-kserve
docker image rm dev.local/acme-fraud-predictor:0.1.0 2>/dev/null || true
kind get clusters
echo "teardown done"

Bash

./scripts/cluster_down.sh

Expected output: kind get clusters prints No kind clusters found., then teardown done.

Optional: Build the Cluster on EKS with Terraform

This appendix creates the same platform on Amazon EKS with Terraform. It costs money from the moment apply finishes until destroy completes, and it was not run or validated for this guide.

Security
Billable resources: the EKS control plane (USD 0.10 per hour on standard support), two t3.large nodes, one NAT gateway, EBS volumes and data transfer. Check the AWS Pricing Calculator for your region before you start, set a Budgets alert, and destroy the same day.

terraform/main.tf:

HCL

# OPTIONAL and NOT FREE: an EKS cluster for the same project. Not run for this guide.
# Costs while it exists: EKS control plane, two EC2 nodes, a NAT gateway, EBS volumes.
# Destroy the same day: terraform destroy
terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.72" }
  }
}
 
provider "aws" {
  region = var.region
  default_tags { tags = { project = "acme-fraud-kserve", owner = var.owner } }   # find every resource later
}
 
variable "region" { default = "ap-south-1" }
variable "owner" { type = string }   # your name or email, required
 
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "6.7.3"
 
  name               = "acme-kserve"
  cidr               = "10.42.0.0/16"
  azs                = ["${var.region}a", "${var.region}b"]
  private_subnets    = ["10.42.1.0/24", "10.42.2.0/24"]
  public_subnets     = ["10.42.101.0/24", "10.42.102.0/24"]
  enable_nat_gateway = true
  single_nat_gateway = true            # one NAT, not one per zone: cheaper, fine for a lab
}
 
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.29.0"
 
  name                                     = "acme-kserve"
  kubernetes_version                       = "1.33"
  endpoint_public_access                   = true
  enable_cluster_creator_admin_permissions = true
  vpc_id                                   = module.vpc.vpc_id
  subnet_ids                               = module.vpc.private_subnets
 
  addons = {
    coredns                = {}
    eks-pod-identity-agent = { before_compute = true }
    kube-proxy             = {}
    vpc-cni                = { before_compute = true }
  }
 
  eks_managed_node_groups = {
    default = {
      ami_type       = "AL2023_x86_64_STANDARD"
      instance_types = ["t3.large"]    # 2 vCPU, 8 GiB each: enough for Istio + Knative + KServe
      min_size       = 2
      max_size       = 2
      desired_size   = 2
    }
  }
}
 
output "kubeconfig_command" {
  value = "aws eks update-kubeconfig --region ${var.region} --name ${module.eks.cluster_name}"
}

Bash

cd terraform
terraform init && terraform apply -var owner="$USER"
$(terraform output -raw kubeconfig_command)

Then two changes from the kind path. Run the KServe installer as in Milestone 1 (skip kind create cluster). And push the image to ECR instead of kind load, use the ECR image URI in the manifests (no dev.local/), and let Knative resolve digests normally. When done:

Bash

kubectl delete isvc acme-fraud -n acme-fraud      # remove load balancers the cluster created
terraform destroy -var owner="$USER"

Common Mistake
Running terraform destroy while Istio's LoadBalancer Service still exists. AWS created that load balancer, not Terraform, so it can block VPC deletion. Delete Kubernetes Services of type LoadBalancer (or uninstall Istio) first.

Present This Project in an Interview

The 60-second version

"I ran a KServe canary on a local kind cluster in Knative mode. One image holds three model versions and an env var picks one, so each revision differs only in the model. I sent 10 percent to v2 and did not trust the status: a script sent 500 real requests through the Istio ingress and counted which version answered, using response headers, against a 3-sigma band. A rule written in advance promoted v2 on error rate and p95. Then I rolled out v3, which had a renamed training feature: same offline score, healthy probes, and a 100 percent error rate on its 10 percent. The rule said rollback, I set canaryTrafficPercent to 0, and real traffic confirmed 0 requests on v3."

Questions to expect

"Why did the readiness probe not catch v3?" The model loads fine; it fails only when a real request lacks the renamed column. Probes check startup, not the feature contract. Canary traffic or a contract test in CI catches it.
"Why count responses instead of reading the traffic field?" The field is intent. Counting real responses proves the ingress, Host routing and revisions all agree.
"Is 49 canary requests enough?" For a 100 percent error rate, yes. For a 0.5 percent regression, no: I would run longer or raise the canary percentage after a clean first window.
"Why Knative mode when Standard is the v0.21 default?" canaryTrafficPercent needs revisions and a weighted route, which Knative provides. Standard mode would need a separate rollout tool.
"How would you automate this?" A pipeline applies the canary, runs traffic_check.py on a schedule, and applies the promote or rollback manifest based on canary_decide.py's exit code, the same pattern as the CI/CD project's gate.

What to show on screen

split_status.sh next to traffic_check.py output for the 90/10 step.
The v3 ROLLBACK decision with the latency check passing and the error check failing.
kubectl get revisions -n acme-fraud showing the full rollout history.

Found a mistake, an outdated step or a broken link?

KServe Canary Deployment on Kubernetes with kind

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem

What you build

The rollout you will run

Repository layout

Prerequisites (by module title)

Versions used

Milestone 1: Create the kind Cluster and Install KServe

Why Knative mode for a canary

Step 1: Write the kind config

Step 2: Create the cluster and install KServe

Step 3: Check the platform

Acceptance check

Milestone 2: Build the Predictor and Load It into kind

Why one image with three versions

Step 1: Train the versions

Step 2: Write the predictor

Step 3: Smoke test it locally before Kubernetes

Step 4: Build and load into kind

Milestone 3: Deploy v1 and Verify It Two Ways

Why two checks

Step 1: Write the manifests

Step 2: Write the verification tools

Step 3: Deploy and check

Acceptance check

Milestone 4: Canary v2 at 10 Percent and Decide

Step 1: Write the decision rules

Step 2: Start the canary

Step 3: Verify with traffic, then decide

Acceptance check

Milestone 5: Promote v2

Step 1: Promote

Acceptance check

Milestone 6: Canary the Broken v3 and Roll Back

Step 1: Start the v3 canary and measure

Step 2: Roll back

Step 3: Leave a clean state

Acceptance check

Break It on Purpose

Step 1: A revision that never becomes ready

Step 2: An image Knative cannot resolve

Step 3: Clean up after breaking

Validation & Testing

Step 1: Final checks

Definition of done

Step 2: Tear down

Optional: Build the Cluster on EKS with Terraform

Present This Project in an Interview

The 60-second version

Questions to expect

What to show on screen