CI/CD for ML with DVC, MLflow and GitHub Actions

Build CI/CD for ML: DVC-versioned data, a trusted metric gate on every pull request, approved MLflow promotion and a tested rollback.

Architecture Overview & Problem Statement

This project builds a CI/CD pipeline for a machine learning model with DVC, MLflow and GitHub Actions. Every pull request retrains the model, and a separate job that never sees the pull request's files judges it with main's code, rules and data: if it is worse than the current champion, the merge is blocked. Every merge registers an exact model version, scores the test set once, and waits for a named reviewer before the champion alias moves. A rollback moves champion back to an earlier exact version through the same approval.

You finish with four pieces of evidence: a blocked pull request, a merged pull request that became a registered version, an approved promotion with the reviewer recorded, and a rollback. Plus one reproducibility proof: the same commit and data version rebuilt the same model file and metrics on the same machine.

The problem

The acme-fraud team at acme-shop retrains its fraud model whenever someone changes a parameter. Last quarter a "small tuning PR" dropped validation PR-AUC by a third and went live because nobody compared it with the model already serving. When the team tried to go back, nobody knew which data and code had produced the previous model. This project fixes all three: gate on metrics before merge, promote exact versions with a recorded approval, and keep enough lineage to rebuild or roll back any version.

How a pull request is judged

The pull request's code runs in one job. A second job checks out only main, downloads nothing from the pull request except the trained model file, and decides.

flowchart TD PR([Pull request]) --> T[Job 1: train
PR code + PR params] T -->|model file only| J[Job 2: judge
main's code, rules, data] J -->|fail| X([Blocked]) J -->|pass| M([Merge allowed]) classDef person fill:#DBEAFE,color:#1E293B,stroke:#326CE5 classDef untrusted fill:#FFEDD5,color:#1E293B,stroke:#F97316 classDef trusted fill:#326CE5,color:#FFFFFF,stroke:#326CE5 classDef bad fill:#EF4444,color:#FFFFFF,stroke:#EF4444 classDef good fill:#10B981,color:#FFFFFF,stroke:#10B981 class PR person class T untrusted class J trusted class X bad class M good

How a merge becomes the champion

stateDiagram-v2 [*] --> Registered: merge to main
gate passed Registered --> Tested: test set once Tested --> Champion: reviewer approves Champion --> FormerChampion: next promotion FormerChampion --> Champion: approved rollback

Where state lives

A free public GitHub repository has no server to host a model registry, so this project keeps state on two extra branches, each checked out as a folder next to the code:

Branch	Folder	Holds
`main`	project root	code, params, `dvc.lock`, data pointer
`dvc-store`	`.dvc-store/`	DVC remote: content-addressed data files
`registry-state`	`.registry/`	MLflow SQLite DB, model artifacts, test reports, champion baseline, audit log

Every job starts a local MLflow server on top of .registry/, and release jobs commit it back. This is a portfolio workaround, not a production registry: the SQLite file and model artifacts grow with every version, Git keeps every old copy forever, and only one writer can work at a time. In a company you point MLFLOW_TRACKING_URI at a shared MLflow server and the DVC remote at S3, and give only main release jobs credentials through OIDC (covered in CI/CD and Continuous Training for ML). The scripts do not change.

How the data is split

Split	Share	Used for
fit	51.2%	training
calibration	12.8%	picking the decision threshold
validation	16%	every gate decision (96 fraud rows)
test	20%	one final report per version (120 fraud rows)

The threshold is chosen on calibration and checked on validation, so the recall gate is an independent check, not a number the model was tuned to hit.

Repository layout

◈ DIAGRAM

acme-fraud-cicd/
├── params.yaml                    # model knobs, split, threshold policy, gate rules
├── requirements.txt               # pinned; also a DVC dependency of training
├── dvc.yaml  dvc.lock             # the train stage and its exact inputs
├── data/payments.csv.dvc          # pointer to the data version (md5)
├── src/acme_fraud/                # data.py (splits + lineage helpers), train.py
├── scripts/
│   ├── make_data.py               # create the dataset once
│   ├── registry.sh                # start/stop MLflow on .registry/ (waits for shutdown)
│   ├── save_registry.sh           # commit + push .registry/, fail loudly
│   ├── registry_util.py           # champion and version lookup
│   ├── evaluate_model.py          # evaluator                 exit 0 / 4
│   ├── gate.py                    # quality gate              exit 0 / 1 / 4
│   ├── register_candidate.py      # exact-version register    exit 0 / 1 / 4
│   ├── test_once.py               # test once, recoverable    exit 0 / 1 / 3 / 4
│   ├── promote_ci.py              # promote / roll back       exit 0 / 1 / 3 / 4 / 5
│   ├── repro_check.py             # rebuild and compare       exit 0 / 1 / 4
│   └── whoserves.py               # what @champion loads now
├── tests/                         # test_gate.py, test_lifecycle.py
└── .github/workflows/             # pr-gate.yml, release.yml

Prerequisites (by module title)

Deploy an ML Model with FastAPI, Docker and AWS (the previous project: same dataset and model family).
Data Versioning with DVC, MLflow Experiment Tracking and Model Registry, CI/CD and Continuous Training for ML, and GitHub Actions.
Tools: Python 3.12, Git 2.40 or newer, the GitHub CLI gh, a free GitHub account, curl.

This project reuses the script names and exit codes from CI/CD and Continuous Training for ML. The versions below are the ones this project was run with, extended with checks found in review. If yours differ, keep the behaviour described here.

Versions used

Tool	Version	Checked
DVC / MLflow	3.67.1 / 3.18.0	PyPI, 5 Nov 2026
scikit-learn / pandas / numpy	1.9.1 / 3.1.1 / 2.5.4	PyPI, 5 Nov 2026
checkout / setup-python / upload / download-artifact	v7.0.1 / v7.1.0 / v7.0.2 / v7.0.0, pinned by SHA	GitHub tags, 5 Nov 2026

Tip
What was run for this guide: every script, the 13 tests, the full local flow (first champion, blocked PR, merged PR, promotion, rollback), four break-it exercises and the reproducibility check from a fresh clone, in one clean run on Python 3.12 in a Linux sandbox. Outputs marked "real run" come from that run. The judge job was simulated locally by evaluating the PR's model file from a main checkout. The two GitHub Actions workflows, the reviewer lookup through the GitHub API, branch protection and environment approval were not run (no GitHub access from the sandbox). The workflows were checked for YAML validity only, which is not the same as running them. Run them once yourself before calling the project done.

Milestone 1: Set Up the Repo, the Data and the DVC Pipeline

This milestone creates the repository, versions the dataset with DVC, and defines one pipeline stage so dvc repro retrains only when code, data, parameters or pinned libraries change.

Why the data needs a version too

A model is a function of code, parameters, libraries and data. Git versions the first three. DVC versions the data by storing the file in a remote and committing a small pointer (payments.csv.dvc, containing the file's md5) to Git. One commit then pins all four.

Step 1: Create the project

Bash

mkdir acme-fraud-cicd && cd acme-fraud-cicd
git init -b main
python3.12 -m venv .venv && source .venv/bin/activate
mkdir -p src/acme_fraud scripts tests .github/workflows
touch src/acme_fraud/__init__.py
export PYTHONPATH=src MLFLOW_DISABLE_AGENT_HINT=1 MLFLOW_DISABLE_TELEMETRY=true

requirements.txt:

TEXT

scikit-learn==1.9.1
pandas==3.1.1
numpy==2.5.4
joblib==1.6.0
pyyaml==6.0.3
mlflow==3.18.0
dvc==3.67.1
pytest==9.2.0

Bash

pip install -r requirements.txt

.gitignore:

TEXT

.venv/
__pycache__/
.pytest_cache/
.registry/
.dvc-store/
.mlflow-server.*
metrics/
candidate/
trusted/
/model

Step 2: Add the parameters, data and training code

params.yaml. The gate block is the rulebook; the judge job reads it from main, so a pull request cannot loosen it. The floors were written before the first run of this design.

YAML

seed: 42
data:
  path: data/payments.csv
  rows: 20000
  fraud_rate: 0.03
split:
  test_size: 0.20      # 20 percent: looked at once per registered version
  val_size: 0.20       # 16 percent overall: every gate decision uses this
  calib_size: 0.20     # 12.8 percent overall: only for picking the threshold
model:
  max_iter: 200
  learning_rate: 0.05
  max_leaf_nodes: 31
threshold:
  calib_recall_target: 0.80    # threshold = highest one reaching this on CALIBRATION
gate:
  # Written before the first run of this design. Illustrative, not business targets.
  min_val_pr_auc: 0.18
  min_val_recall: 0.65         # checked on VALIDATION, which never saw the threshold choice
  max_drop_vs_champion: 0.01

src/acme_fraud/data.py (same generator as the previous project, plus the four-way split and lineage helpers):

Python

"""Payments data and the fixed evaluation layout every script shares."""
import hashlib
import json
from pathlib import Path
 
import numpy as np
import pandas as pd
import yaml
from sklearn.model_selection import train_test_split
 
FEATURES = ["amount", "hour", "merchant_category", "country_mismatch",
            "card_age_days", "txns_last_24h"]
CATEGORIES = ["grocery", "electronics", "travel", "fuel", "fashion", "gaming"]
 
 
def load_params(path: str = "params.yaml") -> dict:
    with open(path) as f:
        return yaml.safe_load(f)
 
 
def make_payments(rows: int, fraud_rate: float, seed: int) -> pd.DataFrame:
    """Synthetic card payments with a weak, noisy fraud signal (same as the previous project)."""
    rng = np.random.default_rng(seed)
    df = pd.DataFrame({
        "txn_id": [f"txn_{i:06d}" for i in range(rows)],
        "amount": np.round(rng.lognormal(mean=3.5, sigma=1.0, size=rows), 2),
        "hour": rng.integers(0, 24, size=rows),
        "merchant_category": rng.choice(CATEGORIES, size=rows),
        "country_mismatch": rng.binomial(1, 0.08, size=rows),
        "card_age_days": rng.integers(1, 3650, size=rows),
        "txns_last_24h": rng.poisson(2.0, size=rows),
    })
    score = (
        0.9 * np.log1p(df["amount"]) / 5
        + 1.2 * df["country_mismatch"]
        + 0.6 * df["merchant_category"].isin(["electronics", "gaming"])
        + 0.5 * ((df["hour"] <= 4) | (df["hour"] >= 23))
        + 0.25 * np.minimum(df["txns_last_24h"], 8) / 2
        - 0.6 * (df["card_age_days"] > 365)
        + rng.normal(0, 1.0, size=rows)
    )
    df["is_fraud"] = (score >= np.quantile(score, 1 - fraud_rate)).astype(int)
    return df
 
 
def load_splits(params: dict):
    """train_fit / calib / val / test = 51.2 / 12.8 / 16 / 20 percent, stratified and seeded."""
    df = pd.read_csv(params["data"]["path"])
    s, seed = params["split"], params["seed"]
    rest, test = train_test_split(df, test_size=s["test_size"], stratify=df["is_fraud"], random_state=seed)
    train, val = train_test_split(rest, test_size=s["val_size"], stratify=rest["is_fraud"], random_state=seed)
    fit, calib = train_test_split(train, test_size=s["calib_size"], stratify=train["is_fraud"], random_state=seed)
    return fit, calib, val, test
 
 
# ---- lineage helpers: what exactly was this model judged on? ----
def file_md5(path: str) -> str:
    return hashlib.md5(Path(path).read_bytes()).hexdigest()
 
 
def dvc_md5(dvc_file: str = "data/payments.csv.dvc") -> str:
    """The data version Git points at (the md5 DVC wrote into the .dvc file)."""
    return yaml.safe_load(Path(dvc_file).read_text())["outs"][0]["md5"]
 
 
def eval_config_sha(params: dict) -> str:
    """Fingerprint of everything that decides WHICH rows are validation and test."""
    cfg = {"seed": params["seed"], "split": params["split"], "data": params["data"]}
    return hashlib.sha256(json.dumps(cfg, sort_keys=True).encode()).hexdigest()[:16]

scripts/make_data.py:

Python

"""Create data/payments.csv once. After this, DVC owns the file's versions."""
from pathlib import Path
 
from acme_fraud.data import load_params, make_payments
 
params = load_params()
path = Path(params["data"]["path"])
path.parent.mkdir(parents=True, exist_ok=True)
make_payments(params["data"]["rows"], params["data"]["fraud_rate"], params["seed"]).to_csv(path, index=False)
print(f"wrote {path}")

src/acme_fraud/train.py:

Python

"""Train the model; pick its threshold on the CALIBRATION split. Run by `dvc repro`.
 
This script does not decide whether the model ships. The judge job runs main's
evaluator and gate on the model this produces.
"""
import hashlib
import json
import os
from pathlib import Path
 
import joblib
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import recall_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
 
from acme_fraud.data import FEATURES, eval_config_sha, file_md5, load_params, load_splits
 
MODEL_DIR = Path("model")
 
 
def build_model(params: dict) -> Pipeline:
    pre = ColumnTransformer(
        [("cat", OneHotEncoder(handle_unknown="ignore"), ["merchant_category"])],
        remainder="passthrough",
    )
    clf = HistGradientBoostingClassifier(
        max_iter=params["model"]["max_iter"],
        learning_rate=params["model"]["learning_rate"],
        max_leaf_nodes=params["model"]["max_leaf_nodes"],
        class_weight="balanced",
        random_state=params["seed"],
    )
    return Pipeline([("pre", pre), ("clf", clf)])
 
 
def pick_threshold(y_true, proba, recall_target: float) -> float:
    """Highest threshold reaching the recall target; 0.0 is the last try."""
    for t in np.round(np.arange(0.99, -0.001, -0.01), 2):
        if recall_score(y_true, proba >= t) >= recall_target:
            return float(t)
    raise ValueError(f"no threshold reaches recall {recall_target}")
 
 
def main() -> None:
    params = load_params()
    fit, calib, _val, _test = load_splits(params)     # val and test are never touched here
    model = build_model(params).fit(fit[FEATURES], fit["is_fraud"])
    calib_proba = model.predict_proba(calib[FEATURES])[:, 1]
    threshold = pick_threshold(calib["is_fraud"], calib_proba, params["threshold"]["calib_recall_target"])
 
    MODEL_DIR.mkdir(exist_ok=True)
    joblib.dump(model, MODEL_DIR / "model.joblib.tmp")
    os.replace(MODEL_DIR / "model.joblib.tmp", MODEL_DIR / "model.joblib")
    meta = {
        "model_sha256": hashlib.sha256((MODEL_DIR / "model.joblib").read_bytes()).hexdigest(),
        "threshold": threshold,
        "trained_on_data_md5": file_md5(params["data"]["path"]),
        "eval_config_sha": eval_config_sha(params),
        "features": FEATURES,
    }
    (MODEL_DIR / "metadata.json.tmp").write_text(json.dumps(meta, indent=2))
    os.replace(MODEL_DIR / "metadata.json.tmp", MODEL_DIR / "metadata.json")
    print(f"trained model sha256={meta['model_sha256'][:12]} threshold={threshold}")
 
 
if __name__ == "__main__":
    main()

Step 3: Define the pipeline

dvc.yaml (note requirements.txt as a dependency: a library upgrade alone must retrain):

YAML

stages:
  train:
    cmd: python -m acme_fraud.train
    deps:
      - src/acme_fraud
      - data/payments.csv
      - requirements.txt          # a library upgrade must retrain too
    params:
      - seed
      - data
      - split
      - model
      - threshold
    outs:
      - model

Step 4: Commit the first version

Bash

git add -A && git commit -m "scaffold"
dvc init
python scripts/make_data.py
dvc add data/payments.csv
dvc repro
git add -A && git commit -m "v1 baseline pipeline"

Expected output of dvc repro (real run, last line):

TEXT

trained model sha256=c609f5f91f7b threshold=0.31

Acceptance check

cat data/payments.csv.dvc shows an md5, and dvc.lock lists the same md5 under the train stage.
Running dvc repro again prints Stage 'train' didn't change, skipping.

Common Mistake
Committing data/payments.csv to Git as well. dvc add writes it into data/.gitignore; if you forced it in before, run git rm --cached data/payments.csv.

Milestone 2: Put the Data Store and the Registry on Branches

This milestone creates the dvc-store and registry-state branches, checks each out as a folder with git worktree, points DVC at one and a local MLflow server at the other, and adds a save script that never hides a failed push.

Step 1: Create the two state branches

Bash

for b in dvc-store registry-state; do
  git switch --orphan "$b" && git commit --allow-empty -m "init $b" && git switch main
done
git worktree add .dvc-store dvc-store
git worktree add .registry registry-state

Step 2: Point DVC at the data store and push the data

Bash

dvc remote add -d store .dvc-store          # relative to where you run it
cat .dvc/config
dvc push
cd .dvc-store && git add -A && git commit -m "data v1" && cd ..
git add .dvc/config && git commit -m "dvc remote: dvc-store branch"

Expected output (real run; the 4 files are the dataset plus the cached model outputs):

TEXT

[core]
    remote = store
['remote "store"']
    url = ../.dvc-store
4 files pushed

Common Mistake
Typing dvc remote add -d store ../.dvc-store because the config file lives in .dvc/. DVC converts the path you type (relative to your shell) into one relative to the config file, so ../.dvc-store becomes ../../.dvc-store, outside your repo, and dvc push silently fills the wrong folder.

Step 3: Start, stop and save the registry

scripts/registry.sh (stop waits until the server and its workers have exited, so nothing is still writing the database):

Bash

#!/usr/bin/env bash
# Start or stop a local MLflow server whose state lives in .registry/
# (.registry/ is a git worktree of the registry-state branch).
# Usage: scripts/registry.sh start|stop
set -euo pipefail
case "${1:-}" in
  start)
    mkdir -p .registry/artifacts
    mlflow server --backend-store-uri sqlite:///.registry/mlflow.db \
      --artifacts-destination ./.registry/artifacts \
      --host 127.0.0.1 --port 5000 > .mlflow-server.log 2>&1 &
    echo $! > .mlflow-server.pid
    for _ in $(seq 1 60); do curl -sf http://127.0.0.1:5000/health >/dev/null && { echo "mlflow up"; exit 0; }; sleep 1; done
    echo "mlflow did not start, see .mlflow-server.log"; exit 1 ;;
  stop)
    # Stop the server AND its workers, and wait until they are gone, so nothing is
    # still writing mlflow.db when we commit it.
    [ -f .mlflow-server.pid ] || { echo "mlflow not running"; exit 0; }
    PID=$(cat .mlflow-server.pid)
    pkill -TERM -P "$PID" 2>/dev/null || true
    kill -TERM "$PID" 2>/dev/null || true
    for _ in $(seq 1 30); do
      if ! kill -0 "$PID" 2>/dev/null && ! pgrep -P "$PID" >/dev/null 2>&1; then
        rm -f .mlflow-server.pid; echo "mlflow stopped"; exit 0
      fi
      sleep 1
    done
    echo "mlflow did not stop within 30 s; not safe to commit the registry"; exit 1 ;;
  *) echo "usage: $0 start|stop"; exit 4 ;;
esac

scripts/save_registry.sh ("nothing to commit" is the only failure it accepts; a failed commit or a rejected push fails the job):

Bash

#!/usr/bin/env bash
# Stop MLflow cleanly, then commit and push .registry/. Fails loudly on any real error.
# Usage: scripts/save_registry.sh "<commit message>"
set -euo pipefail
scripts/registry.sh stop
cd .registry
git add -A
if git diff --cached --quiet; then echo "nothing to save"; exit 0; fi   # the only harmless case
git -c user.name="${GIT_NAME:-release-bot}" -c user.email="${GIT_EMAIL:-release-bot@users.noreply.github.com}" \
  commit -q -m "$1"
if [ "${PUSH:-1}" = "1" ]; then
  # A rejected push means someone else moved the registry. Do not overwrite it: fail, rerun.
  git push origin HEAD:registry-state || { echo "ERROR: push rejected; registry moved. Rerun the job."; exit 1; }
fi
echo "registry saved: $(git log --oneline -1)"

scripts/registry_util.py:

Python

"""Small helpers shared by the release scripts."""
import os
 
from mlflow import MlflowClient
from mlflow.exceptions import MlflowException
 
MODEL_NAME = "acme-fraud"
os.environ.setdefault("MLFLOW_TRACKING_URI", "http://127.0.0.1:5000")
os.environ.setdefault("MLFLOW_ENABLE_ARTIFACTS_PROGRESS_BAR", "false")   # quieter CI logs
 
 
def client() -> MlflowClient:
    return MlflowClient()
 
 
def current_champion(c: MlflowClient):
    """The version behind @champion, or None if no champion exists yet."""
    try:
        return c.get_model_version_by_alias(MODEL_NAME, "champion")
    except MlflowException as err:
        # MLflow 3.18 reports a missing alias as INVALID_PARAMETER_VALUE
        if err.error_code in ("INVALID_PARAMETER_VALUE", "RESOURCE_DOES_NOT_EXIST"):
            return None
        raise
 
 
def get_version(c: MlflowClient, version: str):
    try:
        return c.get_model_version(MODEL_NAME, str(version))
    except MlflowException:
        return None

Bash

chmod +x scripts/*.sh
scripts/registry.sh start          # prints: mlflow up
curl -s localhost:5000/health      # prints: OK

--artifacts-destination makes the server store artifact paths as mlflow-artifacts:/..., relative to the server, instead of absolute paths of your laptop. That is what lets a CI runner with a different home folder read the same registry.

Acceptance check

git worktree list shows three entries: the project, .dvc-store and .registry.
scripts/registry.sh stop prints mlflow stopped and no mlflow server process is left.

Milestone 3: Write the Evaluator and the Gate

This milestone writes the two scripts that decide whether a model may ship: an evaluator that recomputes validation metrics and checks the data on disk is the version Git points at, and a gate that applies fixed floors, requires the same evaluation data as the champion, and compares with the champion.

Where the trust boundary is

In CI the evaluator and gate run in the judge job, which checks out only main. Every input it reads, params.yaml, the data pointer, the data, the libraries and the scripts, comes from main. The only thing it takes from the pull request is the trained model file. The evaluator still takes --params and --dvc-file explicitly, so it never depends on whatever happens to be in the working directory.

What this does not protect against, said plainly:

The judge unpickles the PR's model, and unpickling runs code. A deliberately malicious PR could tamper with the evaluation at load time. That is why the judge job has no secrets and a read-only token.
The PR's training code has the data. It could train on validation rows and inflate its score. Code review before merge catches that; the release job then retrains from reviewed code on main, and the test set is scored once after that.

Step 1: Write the evaluator

All metrics are stored at full precision. Rounding happens only when printing, so a PR-AUC of 0.17996 cannot round up to 0.18 and pass.

scripts/evaluate_model.py:

Python

"""Evaluator: recompute validation metrics for a model file.
 
In CI this runs in the JUDGE job, which checks out only main: main's code, main's
params, main's data pointer and main's data. The PR's files never reach this job;
only the PR's model file does (downloaded as an artifact).
Exit codes: 0 = metrics written, 4 = bad input (missing file, hash or data mismatch).
"""
import argparse
import hashlib
import json
import subprocess
import sys
from pathlib import Path
 
import joblib
from sklearn.metrics import average_precision_score, precision_score, recall_score
 
from acme_fraud.data import FEATURES, dvc_md5, eval_config_sha, file_md5, load_params, load_splits
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--model-dir", default="model")
    ap.add_argument("--params", default="params.yaml")              # explicit, never implied
    ap.add_argument("--dvc-file", default="data/payments.csv.dvc")
    ap.add_argument("--out", default="metrics/val.json")
    a = ap.parse_args()
 
    try:
        params = load_params(a.params)
        meta = json.loads((Path(a.model_dir) / "metadata.json").read_text())
        blob = (Path(a.model_dir) / "model.joblib").read_bytes()
        pointer, actual = dvc_md5(a.dvc_file), file_md5(params["data"]["path"])
    except (FileNotFoundError, KeyError) as err:
        print(f"ERROR: {err}"); return 4
    if hashlib.sha256(blob).hexdigest() != meta["model_sha256"]:
        print("ERROR: model file does not match its metadata"); return 4
    if actual != pointer:
        # the bytes on disk must be the data version Git points at
        print(f"ERROR: {params['data']['path']} does not match {a.dvc_file} (run: dvc checkout --force)"); return 4
 
    _fit, _calib, val, _test = load_splits(params)
    model = joblib.load(Path(a.model_dir) / "model.joblib")   # unpickling runs code: no secrets here
    proba = model.predict_proba(val[FEATURES])[:, 1]
    pred = proba >= meta["threshold"]
    metrics = {   # full precision: decisions use these; round only when printing
        "val_pr_auc": float(average_precision_score(val["is_fraud"], proba)),
        "val_recall": float(recall_score(val["is_fraud"], pred)),
        "val_precision": float(precision_score(val["is_fraud"], pred, zero_division=0)),
        "threshold": meta["threshold"],
        "model_sha256": meta["model_sha256"],
        "data_md5": pointer,                                  # data the model was JUDGED on
        "trained_on_data_md5": meta["trained_on_data_md5"],   # data the model says it trained on
        "eval_config_sha": eval_config_sha(params),
        "git_sha": subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip(),
        "val_rows": int(len(val)), "val_positives": int(val["is_fraud"].sum()),
    }
    Path(a.out).parent.mkdir(parents=True, exist_ok=True)
    Path(a.out).write_text(json.dumps(metrics, indent=2))
    print(f"val_pr_auc={metrics['val_pr_auc']:.4f} val_recall={metrics['val_recall']:.4f} "
          f"threshold={metrics['threshold']} positives={metrics['val_positives']}")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Step 2: Write the gate

The gate's checks, in order:

Two floors from params.yaml: validation PR-AUC and validation recall.
The model was trained on the same data version it is judged on. A pull request that changes the dataset fails here: comparing scores across datasets is apples to oranges, and a dataset change needs its own policy (both models re-scored on one fixed comparison set), which is outside this project.
Same data md5 and same split configuration as the champion's baseline.
Validation PR-AUC no more than 0.01 below the champion's. "Not worse", not "better": with 96 fraud rows in validation, small moves are noise.

scripts/gate.py:

Python

"""Quality gate: fixed floors, same evaluation data as the champion, not worse than it.
 
Exit codes: 0 = pass, 1 = fail, 4 = bad input.
"""
import argparse
import json
import sys
from pathlib import Path
 
import yaml
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--candidate", default="metrics/val.json")
    ap.add_argument("--baseline", default=".registry/baseline/champion.json")
    ap.add_argument("--params", default="params.yaml")
    ap.add_argument("--out", default="metrics/gate.json")
    a = ap.parse_args()
 
    try:
        cand = json.loads(Path(a.candidate).read_text())
        rules = yaml.safe_load(Path(a.params).read_text())["gate"]
    except (FileNotFoundError, KeyError) as err:
        print(f"ERROR: {err}"); return 4
    base = json.loads(Path(a.baseline).read_text()) if Path(a.baseline).exists() else None
 
    checks = [
        ("val_pr_auc >= floor", cand["val_pr_auc"] >= rules["min_val_pr_auc"],
         f"{cand['val_pr_auc']:.4f} vs {rules['min_val_pr_auc']}"),
        ("val_recall >= floor", cand["val_recall"] >= rules["min_val_recall"],
         f"{cand['val_recall']:.4f} vs {rules['min_val_recall']}"),
        ("trained on the evaluation data", cand["trained_on_data_md5"] == cand["data_md5"],
         "dataset changes need the data-change policy, not this gate"
         if cand["trained_on_data_md5"] != cand["data_md5"] else cand["data_md5"][:8]),
    ]
    if base is None:
        checks.append(("champion comparison", True, "no champion yet: first release still needs approval"))
    else:
        same = cand["data_md5"] == base["data_md5"] and cand["eval_config_sha"] == base["eval_config_sha"]
        checks.append(("same evaluation data as champion", same,
                       f"data {cand['data_md5'][:8]} vs {base['data_md5'][:8]}, "
                       f"config {cand['eval_config_sha'][:8]} vs {base['eval_config_sha'][:8]}"))
        limit = base["val_pr_auc"] - rules["max_drop_vs_champion"]
        checks.append(("not worse than champion", same and cand["val_pr_auc"] >= limit,
                       f"{cand['val_pr_auc']:.4f} vs {limit:.4f} (champion v{base['version']} = {base['val_pr_auc']:.4f})"))
 
    passed = all(ok for _, ok, _ in checks)
    for name, ok, detail in checks:
        print(f"{'PASS' if ok else 'FAIL'}  {name}: {detail}")
    print("GATE PASSED" if passed else "GATE FAILED")
    Path(a.out).parent.mkdir(parents=True, exist_ok=True)
    Path(a.out).write_text(json.dumps({"passed": passed, "model_sha256": cand["model_sha256"],
                                       "checks": [{"name": n, "ok": o, "detail": d} for n, o, d in checks]}, indent=2))
    return 0 if passed else 1
 
 
if __name__ == "__main__":
    sys.exit(main())

Step 3: Write the tests

tests/test_gate.py:

Python

"""The gate decides what ships, so test its rules directly."""
import json
import subprocess
import sys
 
CAND = {"model_sha256": "x", "data_md5": "d1", "trained_on_data_md5": "d1", "eval_config_sha": "e1"}
BASE = {"version": "1", "val_pr_auc": 0.30, "data_md5": "d1", "eval_config_sha": "e1"}
 
 
def gate(tmp_path, cand: dict, base: dict | None) -> int:
    (tmp_path / "c.json").write_text(json.dumps({**CAND, **cand}))
    if base:
        (tmp_path / "b.json").write_text(json.dumps(base))
    return subprocess.run([sys.executable, "scripts/gate.py", "--candidate", str(tmp_path / "c.json"),
                           "--baseline", str(tmp_path / "b.json"), "--out", str(tmp_path / "g.json")],
                          capture_output=True).returncode
 
 
def test_first_model_only_needs_floors(tmp_path):
    assert gate(tmp_path, {"val_pr_auc": 0.25, "val_recall": 0.70}, None) == 0
 
 
def test_worse_than_champion_fails(tmp_path):
    assert gate(tmp_path, {"val_pr_auc": 0.28, "val_recall": 0.80}, BASE) == 1
 
 
def test_small_drop_within_tolerance_passes(tmp_path):
    assert gate(tmp_path, {"val_pr_auc": 0.295, "val_recall": 0.80}, BASE) == 0
 
 
def test_recall_below_floor_fails(tmp_path):
    assert gate(tmp_path, {"val_pr_auc": 0.40, "val_recall": 0.60}, None) == 1
 
 
def test_different_evaluation_data_fails(tmp_path):
    assert gate(tmp_path, {"val_pr_auc": 0.40, "val_recall": 0.80, "data_md5": "d2",
                           "trained_on_data_md5": "d2"}, BASE) == 1
 
 
def test_trained_on_other_data_fails(tmp_path):
    assert gate(tmp_path, {"val_pr_auc": 0.40, "val_recall": 0.80, "trained_on_data_md5": "d9"}, None) == 1
 
 
def test_rounding_cannot_lift_a_score_over_the_floor(tmp_path):
    assert gate(tmp_path, {"val_pr_auc": 0.17996, "val_recall": 0.80}, None) == 1   # rounds to 0.1800

tests/test_lifecycle.py (uses a throwaway SQLite registry, no server):

Python

"""Release lifecycle rules: data integrity, promotion refusals, rollback, recovery.
 
Uses a throwaway MLflow SQLite registry in a temp folder; no server needed.
"""
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
 
import mlflow
import pytest
from sklearn.linear_model import LogisticRegression
 
REPO = Path(__file__).resolve().parents[1]
 
 
@pytest.fixture()
def registry(tmp_path):
    uri = f"sqlite:///{tmp_path}/mlflow.db"
    mlflow.set_tracking_uri(uri)
    exp = mlflow.create_experiment("t", artifact_location=(tmp_path / "art").as_uri())
    model = LogisticRegression().fit([[0], [1]], [0, 1])
    with mlflow.start_run(experiment_id=exp):
        info = mlflow.sklearn.log_model(model, name="model")       # one artifact, three versions
    for v, tags in [(1, {"gate": "passed", "tested": "true"}), (2, {"gate": "passed", "tested": "true"}),
                    (3, {"gate": "passed"})]:                       # v3 skipped its test run
        base = {"val_pr_auc": "0.3", "val_recall": "0.8", "threshold": "0.5", "data_md5": "d1",
                "eval_config_sha": "e1", "model_sha256": f"m{v}", "git_sha": f"g{v}"}
        mlflow.register_model(info.model_uri, "acme-fraud", tags={**base, **tags})
    env = {**os.environ, "MLFLOW_TRACKING_URI": uri, "PYTHONPATH": str(REPO / "src")}
    return tmp_path, env
 
 
def promote(reg, *args) -> subprocess.CompletedProcess:
    tmp, env = reg
    return subprocess.run([sys.executable, str(REPO / "scripts/promote_ci.py"), "--baseline-dir",
                           str(tmp / "state/baseline"), "--approved-by", "alice", *args],
                          capture_output=True, text=True, env=env)
 
 
def test_untested_version_cannot_be_promoted(registry):
    assert promote(registry, "--version", "3", "--release-mode", "promote").returncode == 1
 
 
def test_promote_then_rollback_moves_alias_and_baseline(registry):
    tmp, _ = registry
    assert promote(registry, "--version", "1", "--release-mode", "promote").returncode == 0
    assert promote(registry, "--version", "2", "--release-mode", "promote").returncode == 0
    assert promote(registry, "--version", "1", "--release-mode", "rollback").returncode == 0
    base = json.loads((tmp / "state/baseline/champion.json").read_text())
    assert base["version"] == "1" and base["data_md5"] == "d1" and base["eval_config_sha"] == "e1"
    assert len((tmp / "state/audit.log").read_text().splitlines()) == 3
 
 
def test_rollback_needs_a_former_champion(registry):
    assert promote(registry, "--version", "1", "--release-mode", "promote").returncode == 0
    assert promote(registry, "--version", "2", "--release-mode", "rollback").returncode == 1
 
 
def test_stale_approval_is_refused(registry):
    assert promote(registry, "--version", "1", "--release-mode", "promote").returncode == 0
    r = promote(registry, "--version", "2", "--release-mode", "promote", "--expect-champion", "none")
    assert r.returncode == 1 and "Re-approve" in r.stdout
 
 
def test_interrupted_change_blocks_until_recovered(registry):
    tmp, _ = registry
    assert promote(registry, "--version", "1", "--release-mode", "promote").returncode == 0
    pending = {"op_id": "op-test", "mode": "promote", "from": "v1", "to": "2", "approved_by": "alice",
               "requested_by": "bob", "baseline": {"version": "2", "val_pr_auc": 0.3, "val_recall": 0.8,
               "threshold": 0.5, "data_md5": "d1", "eval_config_sha": "e1", "model_sha256": "m2", "git_sha": "g2"}}
    (tmp / "state/pending.json").write_text(json.dumps(pending))          # simulate a crash after step 1
    assert promote(registry, "--version", "2", "--release-mode", "promote").returncode == 5
    assert promote(registry, "--recover").returncode == 0
    assert json.loads((tmp / "state/baseline/champion.json").read_text())["version"] == "2"
    assert not (tmp / "state/pending.json").exists()
 
 
def test_evaluator_rejects_data_that_differs_from_its_pointer(tmp_path):
    shutil.copy(REPO / "data/payments.csv", tmp_path / "p.csv")
    with open(tmp_path / "p.csv", "a") as f:
        f.write("txn_999999,1.0,3,fuel,0,10,1,0\n")                        # one extra row
    params = (REPO / "params.yaml").read_text().replace("path: data/payments.csv", f"path: {tmp_path}/p.csv")
    (tmp_path / "params.yaml").write_text(params)
    r = subprocess.run([sys.executable, "scripts/evaluate_model.py", "--params", str(tmp_path / "params.yaml"),
                        "--out", str(tmp_path / "v.json")], capture_output=True, text=True,
                       env={**os.environ, "PYTHONPATH": "src"}, cwd=REPO)
    assert r.returncode == 4 and "does not match" in r.stdout

Step 4: Run them

Bash

scripts/registry.sh start
python scripts/evaluate_model.py
python scripts/gate.py; echo "exit=$?"
pytest -q tests

Expected output (real run, no champion yet):

TEXT

val_pr_auc=0.2121 val_recall=0.8229 threshold=0.31 positives=96
PASS  val_pr_auc >= floor: 0.2121 vs 0.18
PASS  val_recall >= floor: 0.8229 vs 0.65
PASS  trained on the evaluation data: d526b22f
PASS  champion comparison: no champion yet: first release still needs approval
GATE PASSED
exit=0
13 passed, 1 warning in 49.40s

The first model passes the champion check automatically because there is no champion. It still becomes champion only after a human approves it in the next milestone.

Acceptance check

metrics/val.json contains full-precision metrics, data_md5, trained_on_data_md5 and eval_config_sha.
pytest -q tests shows 13 passed.

Milestone 4: Register, Test Once and Promote the First Champion

This milestone writes the release scripts and runs them by hand once, so the registry has a first champion before any automation exists.

Three rules these scripts enforce

Exact versions. Each step hands the next an exact version number, never "latest" or an alias, because "latest" can change between approval and promotion. Registration is idempotent per commit.
Test once, recoverably. test_once.py checks three data fingerprints (bytes on disk, Git's pointer, the version's tag) and the split configuration, evaluates first, then publishes the report atomically. If the report exists, it refuses a second evaluation but repairs missing registry tags, so a crash between "report written" and "tags written" never blocks the version forever. The test report is the final honest number, not a promotion gate: promotion requires that it exists, not a minimum test score.
Recoverable promotion. promote_ci.py writes an intent file (pending.json) before changing anything, then moves the alias, the baseline, the tag and the audit log in steps that are safe to repeat, verifies alias and baseline, and only then deletes the intent. A crash half way leaves pending.json; every later call exits 5 until --recover finishes the change. --expect-champion refuses if the champion changed since the approver looked.

Step 1: Write the release scripts

scripts/register_candidate.py:

Python

"""Register the gated model as a new exact version. Runs on main only.
 
Exit codes: 0 = registered (or already registered for this commit), 1 = gate not passed, 4 = bad input.
"""
import json
import sys
from pathlib import Path
 
import joblib
import mlflow
import yaml
 
from registry_util import MODEL_NAME, client
 
LINEAGE = ("git_sha", "data_md5", "eval_config_sha", "model_sha256", "threshold", "val_pr_auc", "val_recall")
 
 
def flat(d: dict, prefix: str = "") -> dict:
    out = {}
    for k, v in d.items():
        out.update(flat(v, f"{prefix}{k}.") if isinstance(v, dict) else {f"{prefix}{k}": v})
    return out
 
 
def main() -> int:
    try:
        m = json.loads(Path("metrics/val.json").read_text())
        gate = json.loads(Path("metrics/gate.json").read_text())
    except FileNotFoundError as err:
        print(f"ERROR: {err}"); return 4
    if not gate["passed"] or gate["model_sha256"] != m["model_sha256"]:
        print("REFUSED: gate did not pass for this exact model"); return 1
 
    c = client()
    # Idempotent: one registered version per commit. A rerun returns the same version.
    if c.search_registered_models(filter_string=f"name='{MODEL_NAME}'"):
        for mv in c.search_model_versions(f"name='{MODEL_NAME}'"):
            if mv.tags.get("git_sha") == m["git_sha"]:
                print(f"already registered: {MODEL_NAME} version {mv.version}")
                Path("metrics/registered.json").write_text(json.dumps({"version": mv.version}))
                return 0
 
    mlflow.set_experiment("acme-fraud-release")
    with mlflow.start_run(run_name=f"release-{m['git_sha'][:12]}"):
        mlflow.log_params(flat(yaml.safe_load(Path("params.yaml").read_text())))
        mlflow.log_metrics({k: m[k] for k in ("val_pr_auc", "val_recall", "val_precision")})
        mlflow.log_artifact("metrics/gate.json")
        # MLflow 3.18 saves scikit-learn models with skops, which refuses unknown types.
        # We trust this one type because this job trained the model from reviewed code.
        info = mlflow.sklearn.log_model(
            joblib.load("model/model.joblib"), name="model",
            skops_trusted_types=["sklearn.ensemble._hist_gradient_boosting.predictor.TreePredictor"])
    tags = {k: repr(m[k]) if isinstance(m[k], float) else str(m[k]) for k in LINEAGE}   # repr keeps full precision
    tags["gate"] = "passed"
    mv = mlflow.register_model(info.model_uri, MODEL_NAME, tags=tags)
    c.set_registered_model_alias(MODEL_NAME, "candidate", mv.version)   # informational only
    Path("metrics/registered.json").write_text(json.dumps({"version": mv.version}))
    print(f"registered {MODEL_NAME} version {mv.version} (git {m['git_sha'][:12]}, data {m['data_md5'][:8]})")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Tip
MLflow 3.18 saves scikit-learn models with skops and refuses types it cannot audit. HistGradientBoostingClassifier uses TreePredictor, so log_model fails until you name that one type in skops_trusted_types. Trust a type only because you trained the model yourself, never to silence the error on a file from somewhere else.

scripts/test_once.py:

Python

"""Score the test set once per registered version and record it in the registry.
 
The test report is the final, honest number for a version. It is NOT a promotion
gate: promotion needs the report to exist, not a minimum test score.
Exit codes: 0 = report written, 1 = evaluation failed (nothing written, retry allowed),
            3 = already tested (missing registry tags are repaired), 4 = bad input.
"""
import argparse
import json
import os
import sys
from datetime import datetime, timezone
from pathlib import Path
 
import mlflow
from sklearn.metrics import average_precision_score, recall_score
 
from acme_fraud.data import FEATURES, dvc_md5, eval_config_sha, file_md5, load_params, load_splits
from registry_util import MODEL_NAME, client, get_version
 
REPORTS = Path(".registry/test_reports")
 
 
def tag_from_report(c, v: str, report: dict) -> None:
    """Idempotent: safe to run again after a crash between report and tags."""
    for k in ("test_pr_auc", "test_recall"):
        c.set_model_version_tag(MODEL_NAME, v, k, repr(report[k]))
    c.set_model_version_tag(MODEL_NAME, v, "tested", "true")
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--version", required=True)
    v = ap.parse_args().version
    c = client()
    mv = get_version(c, v)
    if mv is None:
        print(f"ERROR: {MODEL_NAME} version {v} not found"); return 4
    params = load_params()
    pointer, actual = dvc_md5(), file_md5(params["data"]["path"])
    # three-way check: bytes on disk == Git's pointer == what the version was judged on
    if not (actual == pointer == mv.tags.get("data_md5")):
        print(f"ERROR: data mismatch: file {actual[:8]}, pointer {pointer[:8]}, version {mv.tags.get('data_md5', '?')[:8]}"); return 4
    if eval_config_sha(params) != mv.tags.get("eval_config_sha"):
        print("ERROR: split/evaluation config differs from the one this version was judged with"); return 4
 
    REPORTS.mkdir(parents=True, exist_ok=True)
    report_path, lock = REPORTS / f"v{v}.json", REPORTS / f".v{v}.lock"
    if report_path.exists():
        report = json.loads(report_path.read_text())
        if report["version"] != v or report["data_md5"] != actual:
            print(f"ERROR: {report_path} does not belong to version {v} on this data"); return 4
        tag_from_report(c, v, report)
        print(f"REFUSED: version {v} was already tested (registry tags checked and repaired)."); return 3
    try:
        os.close(os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY))
    except FileExistsError:
        print(f"REFUSED: {lock} exists (another run, or a crashed one: check, then delete)."); return 3
    try:
        *_, test = load_splits(params)
        model = mlflow.sklearn.load_model(f"models:/{MODEL_NAME}/{v}")    # exact version, never an alias
        proba = model.predict_proba(test[FEATURES])[:, 1]
        report = {
            "version": v, "data_md5": actual, "eval_config_sha": eval_config_sha(params),
            "test_pr_auc": float(average_precision_score(test["is_fraud"], proba)),
            "test_recall": float(recall_score(test["is_fraud"], proba >= float(mv.tags["threshold"]))),
            "rows": int(len(test)), "positives": int(test["is_fraud"].sum()),
            "evaluated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        }
        tmp = report_path.with_suffix(".json.tmp")
        tmp.write_text(json.dumps(report, indent=2))
        os.link(tmp, report_path)            # publish; fails if the report appeared meanwhile
        tmp.unlink()
    except Exception as err:
        print(f"EVALUATION FAILED, no report written: {err}"); return 1
    finally:
        lock.unlink(missing_ok=True)
    tag_from_report(c, v, report)
    print(f"version {v}: test_pr_auc={report['test_pr_auc']:.4f} test_recall={report['test_recall']:.4f} "
          f"rows={report['rows']} positives={report['positives']}")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

scripts/promote_ci.py:

Python

"""Move the champion alias to an EXACT version, after a human approved it.
 
--release-mode promote : version must have passed the gate and its one test run
--release-mode rollback: version must have been champion before
Every change is written as an intent file first and applied in idempotent steps,
so a crash half way is finished with --recover instead of leaving alias, baseline
and audit log disagreeing.
Exit codes: 0 = done, 1 = refused or verification failed, 3 = already champion,
            4 = bad input, 5 = an unfinished change exists (run --recover).
"""
import argparse
import json
import os
import sys
from datetime import datetime, timezone
from pathlib import Path
 
from registry_util import MODEL_NAME, client, current_champion, get_version
 
 
def write_json(path: Path, data: dict) -> None:
    tmp = path.with_suffix(".tmp")
    tmp.write_text(json.dumps(data, indent=2))
    os.replace(tmp, path)                                   # atomic on one filesystem
 
 
def apply(c, intent: dict, base_dir: Path, pending: Path) -> int:
    """Each step is safe to repeat."""
    c.set_registered_model_alias(MODEL_NAME, "champion", intent["to"])
    write_json(base_dir / "champion.json", intent["baseline"])
    c.set_model_version_tag(MODEL_NAME, intent["to"], "was_champion", "true")
    audit = base_dir.parent / "audit.log"
    line = (f"{intent['op_id']} {intent['mode']} champion {intent['from']} -> v{intent['to']} "
            f"approved_by={intent['approved_by']} requested_by={intent['requested_by']}\n")
    if not (audit.exists() and intent["op_id"] in audit.read_text()):
        with open(audit, "a") as f:
            f.write(line)
    # verify before declaring success
    champ = current_champion(c)
    on_disk = json.loads((base_dir / "champion.json").read_text())
    if champ is None or str(champ.version) != intent["to"] or on_disk["version"] != intent["to"]:
        print("ERROR: verification failed; pending.json kept, run --recover"); return 1
    pending.unlink()
    verb = "promoted" if intent["mode"] == "promote" else "rollback initiated (alias moved)"
    print(f"{verb}: champion {intent['from']} -> v{intent['to']} (verified alias and baseline)")
    return 0
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--version")
    ap.add_argument("--release-mode", choices=["promote", "rollback"])
    ap.add_argument("--baseline-dir", default=".registry/baseline")
    ap.add_argument("--approved-by", help="the person who approved, as verified by the caller")
    ap.add_argument("--requested-by", default="unknown", help="the person or event that asked for it")
    ap.add_argument("--expect-champion", help="refuse if @champion is not this version ('none' = no champion)")
    ap.add_argument("--recover", action="store_true", help="finish an interrupted change")
    a = ap.parse_args()
 
    c = client()
    base_dir = Path(a.baseline_dir); base_dir.mkdir(parents=True, exist_ok=True)
    pending = base_dir.parent / "pending.json"
 
    if a.recover:
        if not pending.exists():
            print("nothing to recover"); return 0
        intent = json.loads(pending.read_text())
        champ = current_champion(c)
        now = f"v{champ.version}" if champ else "none"   # versions compared as strings
        if now not in (intent["from"], f"v{intent['to']}"):
            print(f"ERROR: champion is {now}, neither side of the pending change; resolve by hand"); return 1
        print(f"recovering {intent['op_id']}: {intent['from']} -> v{intent['to']}")
        return apply(c, intent, base_dir, pending)
 
    if not (a.version and a.release_mode and a.approved_by):
        print("ERROR: --version, --release-mode and --approved-by are required"); return 4
    if pending.exists():
        print(f"ERROR: unfinished change in {pending}; run with --recover first"); return 5
    mv = get_version(c, a.version)
    if mv is None:
        print(f"ERROR: {MODEL_NAME} version {a.version} not found"); return 4
    champ = current_champion(c)
    old = f"v{champ.version}" if champ else "none"
    if a.expect_champion and a.expect_champion.lstrip("v") != old.lstrip("v"):
        print(f"REFUSED: champion is {old}, but the approver saw {a.expect_champion}. Re-approve."); return 1
    if champ and str(champ.version) == str(mv.version):
        print(f"version {mv.version} is already champion, nothing to do"); return 3
 
    t = mv.tags
    if a.release_mode == "promote" and not (t.get("gate") == "passed" and t.get("tested") == "true"):
        print(f"REFUSED: version {mv.version} needs gate=passed and tested=true (has {t.get('gate')}, {t.get('tested')})"); return 1
    if a.release_mode == "rollback" and t.get("was_champion") != "true":
        print(f"REFUSED: version {mv.version} was never champion, so it is not a rollback target"); return 1
 
    intent = {
        "op_id": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "mode": a.release_mode, "from": old, "to": str(mv.version),   # str: SQLite and REST stores differ
        "approved_by": a.approved_by, "requested_by": a.requested_by,
        "baseline": {"version": str(mv.version), "val_pr_auc": float(t["val_pr_auc"]),
                     "val_recall": float(t["val_recall"]), "threshold": float(t["threshold"]),
                     "data_md5": t["data_md5"], "eval_config_sha": t["eval_config_sha"],
                     "model_sha256": t["model_sha256"], "git_sha": t["git_sha"]},
    }
    write_json(pending, intent)            # 1. record the intent before changing anything
    return apply(c, intent, base_dir, pending)   # 2. apply, verify, then clear the intent
 
 
if __name__ == "__main__":
    sys.exit(main())

scripts/whoserves.py:

Python

"""Show what a service loading models:/acme-fraud@champion would get right now."""
import mlflow
 
from registry_util import MODEL_NAME, client, current_champion
 
champ = current_champion(client())
if champ is None:
    print("no champion yet")
else:
    model = mlflow.sklearn.load_model(f"models:/{MODEL_NAME}@champion")
    print(f"@champion -> version {champ.version}, git {champ.tags['git_sha'][:12]}, "
          f"model {champ.tags['model_sha256'][:12]}, loaded {type(model).__name__}")

Step 2: Commit, then run the first release by hand

You are the approver for this first release, so you record yourself. In CI the approver comes from GitHub's approval record (Milestone 6).

Bash

git add -A && git commit -m "release scripts"
dvc repro && python scripts/evaluate_model.py && python scripts/gate.py
python scripts/register_candidate.py
python scripts/register_candidate.py          # rerun: same version, no duplicate
python scripts/test_once.py --version 1
python scripts/test_once.py --version 1; echo "exit=$?"
python scripts/promote_ci.py --version 1 --release-mode promote \
  --approved-by "$(git config user.name)" --requested-by "$(git config user.name)" --expect-champion none
python scripts/whoserves.py
cat .registry/baseline/champion.json

Expected output (real run; MLflow's run links trimmed):

Bash

Successfully registered model 'acme-fraud'.
Created version '1' of model 'acme-fraud'.
registered acme-fraud version 1 (git bfe3c4760994, data d526b22f)
already registered: acme-fraud version 1
version 1: test_pr_auc=0.1503 test_recall=0.8417 rows=4000 positives=120
exit=0
REFUSED: version 1 was already tested (registry tags checked and repaired).
exit=3
promoted: champion none -> v1 (verified alias and baseline)
@champion -> version 1, git bfe3c4760994, model c609f5f91f7b, loaded Pipeline
{
  "version": "1",
  "val_pr_auc": 0.21210293312658604,
  "val_recall": 0.8229166666666666,
  "threshold": 0.31,
  "data_md5": "d526b22f540789e8f128b70217bb58ea",
  "eval_config_sha": "ff096e10f9ec3b76",
  "model_sha256": "c609f5f91f7b23ecf9ac91ca99212a356d96891a4183935c3cae291fc0eb18d7",
  "git_sha": "bfe3c4760994c4e70af7228ebddc62bb5c8e65ad"
}

Test PR-AUC (0.150) is below validation (0.212). With 120 fraud rows in test, that gap is within what sampling noise produces; report both numbers with their row counts.

Step 3: Save the registry state and publish everything

Bash

PUSH=0 scripts/save_registry.sh "release v1"    # no remote yet, so commit only
gh repo create acme-fraud-cicd --public --source . --push
git push origin dvc-store registry-state

Acceptance check

.registry/baseline/champion.json names version 1 and includes data_md5 and eval_config_sha.
.registry/audit.log has one promote line with approved_by and requested_by, and no pending.json exists.

Security
The repository is public because branch protection and environment reviewers are free on public repositories. Never put real customer data on dvc-store. This dataset is synthetic.

Milestone 5: Block a Worse Model in a Pull Request

This milestone adds the two-job PR workflow, protects main so the judge check must pass before merging, and opens a pull request that makes the model worse.

Step 1: Write the PR workflow

.github/workflows/pr-gate.yml:

YAML

name: pr-gate
on:
  pull_request:
    branches: [main]
 
permissions:
  contents: read            # read-only token; no secrets anywhere in this workflow
 
concurrency:
  group: pr-gate-${{ github.event.pull_request.number }}
  cancel-in-progress: true
 
jobs:
  # Job 1 runs the PR's code. Its only output is the model file.
  train:
    runs-on: ubuntu-24.04
    steps:
      # Pinned by commit SHA: checkout v7.0.1, setup-python v7.1.0, upload-artifact v7.0.2, download-artifact v7.0.0
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          persist-credentials: false
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: dvc-store
          path: .dvc-store
          persist-credentials: false
      - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: dvc pull data/payments.csv && dvc repro train
        env:
          PYTHONPATH: src
      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
        with:
          name: candidate-model
          path: model/
          retention-days: 7
 
  # Job 2 never checks out the PR. Main's code, rules, data pointer and data judge the model.
  judge:
    needs: train
    runs-on: ubuntu-24.04
    env:
      PYTHONPATH: src
      MLFLOW_DISABLE_AGENT_HINT: "1"
    steps:
      - name: Check out main only (trusted)
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: main
          persist-credentials: false
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: dvc-store
          path: .dvc-store
          persist-credentials: false
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: registry-state
          path: .registry
          persist-credentials: false
      - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt          # main's pinned libraries
      - run: dvc pull data/payments.csv               # main's data version
      - name: Download only the PR's model file
        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131
        with:
          name: candidate-model
          path: candidate/
      - name: Evaluate and gate
        run: |
          python scripts/evaluate_model.py --model-dir candidate --params params.yaml \
            --dvc-file data/payments.csv.dvc --out metrics/val.json
          python scripts/gate.py --params params.yaml --baseline .registry/baseline/champion.json
      - if: always()
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
        with:
          name: gate-${{ github.event.pull_request.number }}
          path: metrics/

Choice	Protects against	Where
Judge checks out only `main`	PR changing scripts, rules, split or data	`judge` job
Model file is the only handoff	PR files reaching the judge	upload / download
`contents: read`, no secrets	A PR stealing credentials	`permissions`
`persist-credentials: false`, SHA pins	Token reuse, moved tags	every checkout

Step 2: Protect main on GitHub

Commit and push the workflow, then in the repository settings:

Branches, add a rule for main: require a pull request, require the status check judge to pass, and do not allow bypassing.
Environments, new environment production: add one or more required reviewers, enable "prevent self-review" if you have a teammate, and limit deployment branches to main.

(Described from GitHub's documentation, not observed for this guide.)

Step 3: Open a pull request that makes the model worse

Bash

git switch -c pr/fewer-trees
sed -i 's/max_iter: 200/max_iter: 10/; s/learning_rate: 0.05/learning_rate: 0.01/' params.yaml
dvc repro
git commit -am "fewer, slower trees"

Judge it locally the way the judge job does: copy the model file out, switch to main, and evaluate with main's files:

Bash

rm -rf ../candidate && cp -r model ../candidate
git switch main && dvc checkout
python scripts/evaluate_model.py --model-dir ../candidate --params params.yaml --dvc-file data/payments.csv.dvc
python scripts/gate.py; echo "exit=$?"

Expected output (real run):

TEXT

trained model sha256=6e073c94ded9 threshold=0.47
val_pr_auc=0.1206 val_recall=0.8750 threshold=0.47 positives=96
FAIL  val_pr_auc >= floor: 0.1206 vs 0.18
PASS  val_recall >= floor: 0.8750 vs 0.65
PASS  trained on the evaluation data: d526b22f
PASS  same evaluation data as champion: data d526b22f vs d526b22f, config ff096e10 vs ff096e10
FAIL  not worse than champion: 0.1206 vs 0.2021 (champion v1 = 0.2121)
GATE FAILED
exit=1

Push the branch and open the pull request (git push -u origin pr/fewer-trees, then gh pr create --fill). On GitHub the judge check should turn red and the merge button stay disabled. Screenshot it, then close the PR.

Acceptance check

The PR shows a failed judge check and cannot be merged.
The gate-<number> artifact contains gate.json with the failed checks.

Common Mistake
Switching the PR workflow to pull_request_target "so it can read secrets". That trigger runs with the base repository's permissions and secrets while you build untrusted code. Keep pull_request and keep secrets out of it.

Milestone 6: Merge a Good Model and Promote It with Approval

This milestone adds the release workflow: a merge to main retrains, gates, registers an exact version and scores the test set once; the promote job then waits for a reviewer, reads who approved from GitHub's record, and moves champion only if it is still the version the approver saw.

Step 1: Write the release workflow

.github/workflows/release.yml:

YAML

name: release
on:
  push:
    branches: [main]          # every merge: retrain, gate, register, test once
  workflow_dispatch:          # someone with write access asks for a promote or rollback
    inputs:
      mode:
        type: choice
        options: [promote, rollback]
        required: true
      version:
        description: Exact registry version number, for example 2
        type: string
        required: true
 
permissions:
  contents: read
 
concurrency:
  group: registry             # one writer of the registry branch at a time
  cancel-in-progress: false
  queue: max                  # queue up to 100 runs; without this a newer pending run replaces an older one
 
env:
  MLFLOW_DISABLE_AGENT_HINT: "1"
  PYTHONPATH: src
 
jobs:
  register:
    if: github.event_name == 'push'
    runs-on: ubuntu-24.04
    permissions:
      contents: write         # to push the registry-state branch
    outputs:
      version: ${{ steps.reg.outputs.version }}
      champion: ${{ steps.reg.outputs.champion }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          persist-credentials: false
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: dvc-store
          path: .dvc-store
          persist-credentials: false
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: registry-state
          path: .registry       # credentials kept here only, to push registry state
      - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: dvc pull data/payments.csv && dvc repro train
      - run: scripts/registry.sh start
      - name: Evaluate, gate, register exact version, test once
        id: reg
        run: |
          python scripts/evaluate_model.py
          python scripts/gate.py
          python scripts/register_candidate.py
          VERSION=$(python -c "import json; print(json.load(open('metrics/registered.json'))['version'])")
          CHAMPION=$(python -c "import json,os; p='.registry/baseline/champion.json'; print(json.load(open(p))['version'] if os.path.exists(p) else 'none')")
          echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
          echo "champion=${CHAMPION}" >> "$GITHUB_OUTPUT"
          python scripts/test_once.py --version "$VERSION" || [ $? -eq 3 ]   # 3 = already tested on a rerun
          {
            echo "### acme-fraud v${VERSION} (current champion: ${CHAMPION})"
            echo '```json'; cat metrics/gate.json; cat ".registry/test_reports/v${VERSION}.json"; echo '```'
          } >> "$GITHUB_STEP_SUMMARY"
      - name: Save registry state (fails the job if the push fails)
        if: always()
        run: scripts/save_registry.sh "register ${GITHUB_SHA::12}"
 
  promote:
    needs: register
    runs-on: ubuntu-24.04
    environment: production    # a required reviewer must approve before this job starts
    permissions:
      contents: write
      actions: read            # to read who approved this run
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          persist-credentials: false
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: registry-state
          path: .registry
      - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Find the reviewer who approved production for this run
        id: who
        env:
          GH_TOKEN: ${{ github.token }}
          RUN_ID: ${{ github.run_id }}
        run: |
          REVIEWER=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${RUN_ID}/approvals" \
            --jq '[.[] | select(.state=="approved") | select(any(.environments[]; .name=="production")) | .user.login] | last // empty')
          [ -n "$REVIEWER" ] || { echo "no recorded approval for production"; exit 1; }
          echo "reviewer=${REVIEWER}" >> "$GITHUB_OUTPUT"
      - run: scripts/registry.sh start
      - name: Promote the exact version that was approved
        env:
          VERSION: ${{ needs.register.outputs.version }}
          EXPECTED: ${{ needs.register.outputs.champion }}
          REVIEWER: ${{ steps.who.outputs.reviewer }}
          REQUESTER: ${{ github.triggering_actor }}
        run: |
          # exit 3 = already champion (a rerun): nothing to do, not a failure
          python scripts/promote_ci.py --version "$VERSION" --release-mode promote \
            --approved-by "$REVIEWER" --requested-by "$REQUESTER" --expect-champion "$EXPECTED" || [ $? -eq 3 ]
      - name: Save registry state (fails the job if the push fails)
        if: always()
        run: scripts/save_registry.sh "promote v${{ needs.register.outputs.version }}"
 
  manual:
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-24.04
    environment: production
    permissions:
      contents: write
      actions: read
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          persist-credentials: false
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: registry-state
          path: .registry
      - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Find the reviewer who approved production for this run
        id: who
        env:
          GH_TOKEN: ${{ github.token }}
          RUN_ID: ${{ github.run_id }}
        run: |
          REVIEWER=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${RUN_ID}/approvals" \
            --jq '[.[] | select(.state=="approved") | select(any(.environments[]; .name=="production")) | .user.login] | last // empty')
          [ -n "$REVIEWER" ] || { echo "no recorded approval for production"; exit 1; }
          echo "reviewer=${REVIEWER}" >> "$GITHUB_OUTPUT"
      - run: scripts/registry.sh start
      - name: Promote or roll back an exact version
        env:                    # inputs go through env, never pasted into the script
          MODE: ${{ inputs.mode }}
          VERSION: ${{ inputs.version }}
          REVIEWER: ${{ steps.who.outputs.reviewer }}
          REQUESTER: ${{ github.triggering_actor }}
        run: |
          [[ "$VERSION" =~ ^[0-9]+$ ]] || { echo "version must be a number"; exit 4; }
          python scripts/promote_ci.py --version "$VERSION" --release-mode "$MODE" \
            --approved-by "$REVIEWER" --requested-by "$REQUESTER"
      - name: Save registry state (fails the job if the push fails)
        if: always()
        env:
          MODE: ${{ inputs.mode }}
          VERSION: ${{ inputs.version }}
        run: scripts/save_registry.sh "${MODE} to v${VERSION}"

Four details:

Queueing. concurrency with cancel-in-progress: false alone keeps one running and one pending run; a third run replaces the pending one, which would silently skip a merge. queue: max queues up to 100 runs in order (GitHub changelog, 7 May 2026).
Who approved. github.actor and github.triggering_actor are who started the run, not who approved it. The promote job reads the approval record from GET /repos/{owner}/{repo}/actions/runs/{run_id}/approvals and fails if no production approval is found. requested_by and approved_by are stored separately. This API call was not run for this guide; check the field names against the GitHub REST docs before relying on it.
Stale approvals. register records the champion at the time the summary was written, and promote passes it as --expect-champion. If someone promoted in between, the promotion is refused and needs a fresh approval.
Inputs through env:. inputs.version is never pasted into the script text, so a value like 2; curl evil cannot run, and the script checks it is a number.

Step 2: Open a good pull request and merge it

Bash

git switch main && git switch -c pr/smaller-leaves
grep max_leaf params.yaml
sed -i 's/max_leaf_nodes: 31/max_leaf_nodes: 15/' params.yaml
grep max_leaf params.yaml
dvc repro && git commit -am "smaller trees: max_leaf_nodes 15"
rm -rf ../candidate && cp -r model ../candidate && git switch main && dvc checkout
python scripts/evaluate_model.py --model-dir ../candidate && python scripts/gate.py; echo "exit=$?"

Expected output (real run):

TEXT

  max_leaf_nodes: 31
  max_leaf_nodes: 15
trained model sha256=68494a4fe9f4 threshold=0.34
val_pr_auc=0.2272 val_recall=0.8333 threshold=0.34 positives=96
PASS  val_pr_auc >= floor: 0.2272 vs 0.18
PASS  val_recall >= floor: 0.8333 vs 0.65
PASS  trained on the evaluation data: d526b22f
PASS  same evaluation data as champion: data d526b22f vs d526b22f, config ff096e10 vs ff096e10
PASS  not worse than champion: 0.2272 vs 0.2021 (champion v1 = 0.2121)
GATE PASSED
exit=0

Validation PR-AUC moved by 0.015 on 96 fraud rows. Claim "not worse", not "better".

Bash

git push -u origin pr/smaller-leaves && gh pr create --fill # wait for a green judge, then merge

Step 3: Approve the promotion

After the merge, the release run starts. register writes a job summary with gate.json, the test report and the current champion. promote then shows "Waiting for review". Read the summary, click Review deployments, tick production, and approve.

The same release, run locally for this guide after git merge --no-ff pr/smaller-leaves (real run):

Bash

Stage 'train' is cached - skipping run, checking out outputs
val_pr_auc=0.2272 val_recall=0.8333 threshold=0.34 positives=96
GATE PASSED
Created version '2' of model 'acme-fraud'.
registered acme-fraud version 2 (git 855754cc6b8d, data d526b22f)
version 2: test_pr_auc=0.1599 test_recall=0.8833 rows=4000 positives=120
exit=0
promoted: champion v1 -> v2 (verified alias and baseline)
exit=0
@champion -> version 2, git 855754cc6b8d, model 68494a4fe9f4, loaded Pipeline

The promote command was promote_ci.py --version 2 --release-mode promote --approved-by reviewer-a --requested-by kushagra --expect-champion 1, standing in for what the workflow passes.

Acceptance check

The release run shows a named approval before promote ran, and the audit line names that reviewer as approved_by.
On the registry-state branch, baseline/champion.json names version 2.

Common Mistake
Promoting @candidate or "the latest version" instead of the number the approver saw. Between approval and promotion another merge can register version 3, and you would ship a model nobody reviewed.

Milestone 7: Roll Back to the Previous Champion

This milestone moves champion back to version 1 through the same approved path, and shows what a rollback does and does not change.

What a rollback here does, and does not do

promote_ci.py --release-mode rollback moves the champion alias and the baseline (including its data md5 and split fingerprint) back to an exact version that was champion before. It prints "rollback initiated (alias moved)" on purpose:

A service that loads models:/acme-fraud@champion serves the old model only after it reloads or restarts.
The version carries references to its code and data (git_sha, data_md5), not the code and data themselves. They tell you exactly what to check out to rebuild it; they do not change any running service's code.

Step 1: Run the rollback

On GitHub: Actions, release, Run workflow, mode rollback, version 1, then approve production. Locally:

Bash

scripts/registry.sh start
python scripts/promote_ci.py --version 1 --release-mode rollback --approved-by reviewer-a --requested-by kushagra; echo "exit=$?"
python scripts/whoserves.py
python scripts/promote_ci.py --version 1 --release-mode rollback --approved-by reviewer-a --requested-by kushagra; echo "exit=$?"
cat .registry/audit.log

Expected output (real run):

Bash

rollback initiated (alias moved): champion v2 -> v1 (verified alias and baseline)
exit=0
@champion -> version 1, git bfe3c4760994, model c609f5f91f7b, loaded Pipeline
version 1 is already champion, nothing to do
exit=3
2026-11-05T04:17:41+00:00 promote champion none -> v1 approved_by=kushagra requested_by=kushagra
2026-11-05T04:18:29+00:00 promote champion v1 -> v2 approved_by=reviewer-a requested_by=kushagra
2026-11-05T04:18:34+00:00 rollback champion v2 -> v1 approved_by=reviewer-a requested_by=kushagra

The baseline followed the alias, so the next pull request is compared with version 1 again.

Acceptance check

whoserves.py shows version 1, and baseline/champion.json names version 1 with its own data_md5.
PUSH=0 scripts/save_registry.sh "rollback" (or the workflow) committed the change.

Tip
In a real incident, a second environment such as rollback with one on-call reviewer gets you back faster than the full release approval. Keep the same script; change only who must approve.

Milestone 8: Prove a Version Is Reproducible

This milestone rebuilds version 2 from a fresh clone at its commit with its data version, and checks that the model file and every registered metric come out identical.

Step 1: Write the check

scripts/repro_check.py:

Python

"""Prove a registered version is reproducible: same commit + same data = same model and metrics.
 
Exit codes: 0 = reproduced exactly, 1 = mismatch, 4 = wrong commit or data checked out.
"""
import argparse
import json
import subprocess
import sys
from pathlib import Path
 
from acme_fraud.data import dvc_md5
from registry_util import client, get_version
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--version", required=True)
    v = ap.parse_args().version
    mv = get_version(client(), v)
    head = subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()
    if mv is None or mv.tags["git_sha"] != head or mv.tags["data_md5"] != dvc_md5():
        print(f"ERROR: check out commit {mv.tags['git_sha'][:12] if mv else '?'} and run dvc pull first"); return 4
 
    subprocess.run(["dvc", "repro", "--force", "train"], check=True, capture_output=True)   # retrain from scratch
    subprocess.run([sys.executable, "scripts/evaluate_model.py", "--out", "metrics/repro.json"],
                   check=True, capture_output=True)
    new = json.loads(Path("metrics/repro.json").read_text())
    ok = True
    for key in ("model_sha256", "val_pr_auc", "val_recall", "threshold", "eval_config_sha"):
        rebuilt = repr(new[key]) if isinstance(new[key], float) else str(new[key])
        same = rebuilt == mv.tags[key]
        ok &= same
        print(f"{'SAME' if same else 'DIFF'}  {key}: registered={mv.tags[key][:12]} rebuilt={rebuilt[:12]}")
    print("REPRODUCED (on this machine and library set)" if ok else "NOT REPRODUCED")
    return 0 if ok else 1
 
 
if __name__ == "__main__":
    sys.exit(main())

Step 2: Rebuild from nothing

Bash

cd /tmp && git clone https://github.com/<you>/acme-fraud-cicd repro && cd repro
git worktree add .dvc-store origin/dvc-store
git checkout <git_sha tag of version 2>
dvc pull
export PYTHONPATH=src
python scripts/repro_check.py --version 2      # the MLflow server from your project folder is still running

Expected output (real run, from a fresh clone on the same machine):

TEXT

SAME  model_sha256: registered=68494a4fe9f4 rebuilt=68494a4fe9f4
SAME  val_pr_auc: registered=0.2272228002 rebuilt=0.2272228002
SAME  val_recall: registered=0.8333333333 rebuilt=0.8333333333
SAME  threshold: registered=0.34 rebuilt=0.34
SAME  eval_config_sha: registered=ff096e10f9ec rebuilt=ff096e10f9ec
REPRODUCED (on this machine and library set)
exit=0

Byte-identical model files depend on the same library versions and CPU family. On another machine the metrics usually match closely even when the file hash differs. Report what you observed and where.

Break It on Purpose

These exercises break the exact things the pipeline trusts. Each ran for this guide; outputs are real.

Step 1: Corrupt the data file the pipeline reads

Bash

echo "txn_999999,1.0,3,fuel,0,10,1,0" >> data/payments.csv
dvc status
python scripts/evaluate_model.py; echo "exit=$?"
dvc checkout --force data/payments.csv.dvc
python scripts/evaluate_model.py; echo "exit=$?"

Expected output:

TEXT

train:
	changed deps:
		modified:           data/payments.csv
ERROR: data/payments.csv does not match data/payments.csv.dvc (run: dvc checkout --force)
exit=4
val_pr_auc=0.2272 val_recall=0.8333 threshold=0.34 positives=96
exit=0

test_once.py runs the same three-way check, so a modified CSV can never produce a test report either.

Step 2: Make a pull request edit the gate

Bash

git switch -c pr/sneaky
sed -i 's/max_iter: 200/max_iter: 10/; s/learning_rate: 0.05/learning_rate: 0.01/' params.yaml
sed -i 's/    passed = all(ok for _, ok, _ in checks)/    passed = True  # "temporary"/' scripts/gate.py
dvc repro && python scripts/evaluate_model.py > /dev/null
python scripts/gate.py | tail -1                      # the PR's own, edited gate
git commit -qam sneaky && rm -rf ../candidate && cp -r model ../candidate
git switch main && dvc checkout
python scripts/evaluate_model.py --model-dir ../candidate > /dev/null
python scripts/gate.py | tail -3                      # what the judge job runs
git branch -D pr/sneaky

Expected output:

TEXT

GATE PASSED
PASS  same evaluation data as champion: data d526b22f vs d526b22f, config ff096e10 vs ff096e10
FAIL  not worse than champion: 0.1165 vs 0.2021 (champion v1 = 0.2121)
GATE FAILED

The PR's gate says yes. The judge never sees the PR's gate.py, params.yaml or data pointer, so it says no.

Step 3: Try to promote a version that skipped the test run

Bash

git commit --allow-empty -m "rebuild for v3"
python scripts/evaluate_model.py > /dev/null && python scripts/gate.py > /dev/null
python scripts/register_candidate.py
python scripts/promote_ci.py --version 3 --release-mode promote --approved-by me; echo "exit=$?"
python scripts/promote_ci.py --version 3 --release-mode rollback --approved-by me; echo "exit=$?"

Expected output:

Bash

registered acme-fraud version 3 (git a58cd3c524e6, data d526b22f)
REFUSED: version 3 needs gate=passed and tested=true (has passed, None)
exit=1
REFUSED: version 3 was never champion, so it is not a rollback target
exit=1

Step 4: Lose a registry tag and recover it without a second test run

Simulate a crash between "test report written" and "tags written" by deleting the tested tag of version 2:

Bash

python -c "import sys; sys.path.insert(0, 'scripts'); from registry_util import client; client().delete_model_version_tag('acme-fraud', '2', 'tested')"
python scripts/promote_ci.py --version 2 --release-mode promote --approved-by me; echo "exit=$?"
python scripts/test_once.py --version 2; echo "exit=$?"

Expected output:

TEXT

REFUSED: version 2 needs gate=passed and tested=true (has passed, None)
exit=1
REFUSED: version 2 was already tested (registry tags checked and repaired).
exit=3

The test set was not scored again, and the version is promotable again. The same idea for promotion, an interrupted change finished with --recover, is covered by test_interrupted_change_blocks_until_recovered.

Validation & Testing

Run the checks below, then tear down. Nothing in this project costs money.

Step 1: Final checks

Bash

pytest -q tests                                  # 13 passed
git worktree list                                # project, .dvc-store, .registry
cat .registry/audit.log                          # promote, promote, rollback
ls .registry/pending.json 2>/dev/null || echo "no unfinished change"
git -C .registry log --oneline | head -5         # every registry change is a commit

Definition of done

Data is tracked by DVC, requirements.txt is a training dependency, and the remote is the dvc-store branch.
The PR workflow has two jobs; the judge job checks out only main and receives only the model file.
One PR was blocked by the judge, and the screenshot is in the README.
One PR was merged, registered as an exact version, tested once, and promoted after an approval whose reviewer is recorded as approved_by.
One rollback moved champion and the baseline (with its data and split fingerprints) to an earlier exact version.
repro_check.py printed REPRODUCED from a fresh clone, and the README says on which machine.
The README says which steps ran on GitHub and which only locally.

Step 2: Tear down

Bash

scripts/registry.sh stop
git worktree remove .dvc-store && git worktree remove .registry
deactivate

To remove everything on GitHub, delete the repository in its settings. Nothing else was created.

Present This Project in an Interview

The 60-second version

"I built CI/CD for a fraud model with DVC, MLflow and GitHub Actions. A pull request's code trains in one job; a second job checks out only main and judges the model file with main's evaluator, rules, split and data, so a PR cannot change how it is graded. The gate needs the same data version as the champion, uses full-precision metrics, and the recall check is independent because the threshold is picked on a separate calibration split. A worse PR, validation PR-AUC 0.121 against a champion at 0.212, was blocked. A merged PR becomes an exact registry version, the test set is scored once, and promotion waits for a reviewer whose name comes from GitHub's approval record. Promotions and rollbacks are written as an intent first, so a crash is recovered, not left half done. On my machine, a fresh clone at the version's commit and data version rebuilt a byte-identical model."

Questions to expect

"Can the PR still cheat?" Yes, two ways: its model file is unpickled by the judge, and its training code could train on validation rows. That is why the judge has no secrets, merges need review, and the release retrains from reviewed code before the one test run.
"Why must the data match the champion's?" Otherwise you compare scores from two datasets. A dataset change needs a different policy: re-score both models on one fixed comparison set.
"Why a tolerance instead of 'must be better'?" With 96 fraud rows in validation, a 0.01 move is noise. Strict improvement would reward luck and block harmless refactors.
"What if two merges land close together?" queue: max runs them in order, one registry writer at a time, and --expect-champion refuses a promotion if the champion changed after approval.
"Is your rollback instant?" Moving the alias is. Serving changes when the service reloads @champion, so in production I would pair it with a reload and check which version is answering.
"What would you change in a company?" A shared MLflow server and an S3 DVC remote instead of branches, OIDC credentials only on main jobs, and a faster rollback environment for on-call.

What to show on screen

The blocked PR with its red judge check.
The release run paused on "Waiting for review", then the audit line with the reviewer.
audit.log and whoserves.py before and after the rollback.

Found a mistake, an outdated step or a broken link?

CI/CD for ML with DVC, MLflow and GitHub Actions

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem

How a pull request is judged

How a merge becomes the champion

Where state lives

How the data is split

Repository layout

Prerequisites (by module title)

Versions used

Milestone 1: Set Up the Repo, the Data and the DVC Pipeline

Why the data needs a version too

Step 1: Create the project

Step 2: Add the parameters, data and training code

Step 3: Define the pipeline

Step 4: Commit the first version

Acceptance check

Milestone 2: Put the Data Store and the Registry on Branches

Step 1: Create the two state branches

Step 2: Point DVC at the data store and push the data

Step 3: Start, stop and save the registry

Acceptance check

Milestone 3: Write the Evaluator and the Gate

Where the trust boundary is

Step 1: Write the evaluator

Step 2: Write the gate

Step 3: Write the tests

Step 4: Run them

Acceptance check

Milestone 4: Register, Test Once and Promote the First Champion

Three rules these scripts enforce

Step 1: Write the release scripts

Step 2: Commit, then run the first release by hand

Step 3: Save the registry state and publish everything

Acceptance check

Milestone 5: Block a Worse Model in a Pull Request

Step 1: Write the PR workflow

Step 2: Protect main on GitHub

Step 3: Open a pull request that makes the model worse

Acceptance check

Milestone 6: Merge a Good Model and Promote It with Approval

Step 1: Write the release workflow

Step 2: Open a good pull request and merge it

Step 3: Approve the promotion

Acceptance check

Milestone 7: Roll Back to the Previous Champion

What a rollback here does, and does not do

Step 1: Run the rollback

Acceptance check

Milestone 8: Prove a Version Is Reproducible

Step 1: Write the check

Step 2: Rebuild from nothing

Break It on Purpose

Step 1: Corrupt the data file the pipeline reads

Step 2: Make a pull request edit the gate

Step 3: Try to promote a version that skipped the test run

Step 4: Lose a registry tag and recover it without a second test run

Validation & Testing

Step 1: Final checks

Definition of done

Step 2: Tear down

Present This Project in an Interview

The 60-second version

Questions to expect

What to show on screen