Learn model drift detection with Evidently: log predictions, measure data and score drift, export to Prometheus and alert before labels arrive.
Saturday, the first evening of acme-shop's festive sale. Traffic is four times normal.
Monitoring everything produces alerts nobody reads. Monitoring the right few things in layers gives you early warnings you trust.
You cannot measure drift on data you did not keep.
A drift detector compares two samples, a reference window and a current window, and turns the difference into a number.
Evidently is an open-source Python library for evaluating and monitoring ML models.
Fraud labels arrive late: a chargeback can take 30 to 90 days. A monitoring plan that waits for labels finds out about the festive sale in February.
Saturday, the first evening of acme-shop's festive sale. Traffic is four times normal.
Monitoring everything produces alerts nobody reads.
You cannot measure drift on data you did not keep.
A drift detector compares two samples, a reference window and a current window, and turns the difference into a number.
Evidently is an open-source Python library for evaluating and monitoring ML models.
Fraud labels arrive late: a chargeback can take 30 to 90 days.
The Monitoring and Logging module covered Prometheus, PromQL, Grafana and Alertmanager.
A drift alert is the start of an investigation, not a decision.
This lab replays the opening incident on your laptop.
Quick reference Common mistakes Watching only service health is how the festive sale went unnoticed.
Roles that use the skills in this module.
AI Engineer
MLOps Engineer
Platform Engineer
Data Engineer
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetData drift means the inputs changed, for example payment amounts doubled during a sale. Concept drift means the relationship between inputs and the right answer changed, for example fraudsters switched tactics so the same features now mean something different. Data drift is visible immediately from the inputs, while concept drift usually only shows once labels arrive.
Watch what you can see today: input drift, the distribution of prediction scores, and the share of each decision, such as how many payments get blocked. Then join labels to logged predictions when they arrive and compute real precision and recall for that older window. Proxy metrics warn early; delayed labels tell you whether the warning mattered.
Not automatically. Drift says the data changed, not that the model got worse. First rule out a pipeline bug, then check whether performance on recent labelled data actually dropped. Retrain when drift is real and performance suffers or is likely to, which the Automated Retraining and ML Incident Response module covers.
Start with the defaults of your tool and a stable, well-known measure such as PSI for the prediction score. Then tune thresholds on your own history: replay several normal weeks and set alerts above the drift values those weeks produce. A threshold that fires every normal Monday will be ignored within a month.