Learn to decide when to retrain, build a gated Airflow retraining pipeline, roll back bad models automatically and run ML incidents with runbooks.
The Diwali sale starts at midnight. By Saturday noon the acme-fraud model is missing a new pattern: fraudsters are buying gift vouchers in small...
The pipeline is a chain of small steps, each with one job and one clear outcome.
A model is judged on labels, and labels are late.
Retraining produces a new model every time. Validation decides whether it deserves to exist. The CI/CD module built the gate.
Everything so far decides whether a model should exist. Release decides who gets it, how fast, and how quickly it can be taken back.
Models fail in four main ways. The skill is telling them apart quickly, because each one has a different fix, and the wrong fix, such as retraining...
The Diwali sale starts at midnight. By Saturday noon the acme-fraud model is missing a new pattern: fraudsters are...
The pipeline is a chain of small steps, each with one job and one clear outcome.
A model is judged on labels, and labels are late.
Retraining produces a new model every time. Validation decides whether it deserves to exist.
Everything so far decides whether a model should exist.
Models fail in four main ways. The skill is telling them apart quickly, because each one has a different fix, and the...
A postmortem is a written review after an incident.
You will simulate a drift alert, let the policy decide from real inputs, retrain on a finalised window with frozen...
Quick reference Common mistakes The first mistake is treating drift as a verdict.
Roles that use the skills in this module.
AI Engineer
MLOps Engineer
Platform Engineer
Data Engineer
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetNo. Drift means the inputs changed, not that the model got worse. Treat drift as a reason to investigate and to train a challenger, then let the validation gate decide whether it is better. Retraining on drift alone and promoting automatically is how teams ship a worse model faster.
Because the labels for recent rows are incomplete. In fraud, many chargebacks arrive weeks later, so recent fraud looks like honest traffic. A model trained on that learns to approve fraud. Only train on rows whose labels have had time to mature.
Each retraining run uses a newer validation window, so the old stored scores describe different rows. Scoring both models on the same new rows is the only fair comparison. The stored baseline is for a fixed evaluation set, not for a moving one.
For routine refreshes of a model whose mistakes are cheap and reversible, with a strong gate, a limit on how often it can happen, and an automatic rollback watching live guardrails. Anything triggered by an anomaly, or any model with costly mistakes such as fraud, should keep a human approval step.