Learn to automate model training with Airflow 3: an ML DAG, artifact passing, repeatable runs, retries, a quality gate, and candidate registration.
It is the week before the festive sale at acme-shop. Someone asks a simple question: when was the fraud model last retrained?
A good ML DAG is short and boring to read. It names the steps and the order. The real work lives in the project you already tested.
Tasks need to hand each other things: a data fingerprint, a folder with a model, a set of metrics.
The DAG says what to run. Another question is where the code runs, and the answer changes what you must install and how much memory you can use.
In the ETL module you learned that a task is idempotent when running it twice gives the same end state as running it once.
A pipeline that runs only when someone remembers is the problem you started with. Airflow can start the DAG in two main ways.
It is the week before the festive sale at acme-shop.
A good ML DAG is short and boring to read. It names the steps and the order.
Tasks need to hand each other things: a data fingerprint, a folder with a model, a set of metrics.
The DAG says what to run. Another question is where the code runs, and the answer changes what you must install and how...
In the ETL module you learned that a task is idempotent when running it twice gives the same end state as running it...
A pipeline that runs only when someone remembers is the problem you started with.
This is the topic the whole module is built for.
The last task records the approved run so that other systems can find it.
An ML pipeline can use far more of the machine than a data pipeline.
You will run Airflow in Docker, mount the project from Reproducible ML Training and Project Structure, and run the...
Quick reference Common mistakes Copying a trained model straight to production in the DAG, with no gate, is how the...
Roles that use the skills in this module.
AI Engineer
MLOps Engineer
Platform Engineer
Data Engineer
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetA data pipeline produces a table, and a model pipeline produces a model file plus the numbers that say whether it is any good. A table that loads is usually fine. A model that trains is not necessarily fine, so a model pipeline needs a decision step that can stop a bad result from being released.
Airflow retries any exception by default, so on its own it cannot tell a full disk from a missing column. Use a retry policy to fail fast on errors that never heal, such as bad data or bad settings, and to retry machine and storage trouble. Set retries to zero on validation and on the quality gate.
Not for the first version. Training inside a worker is simple and fine for small models. Move training into its own container or an external job when it needs different libraries, a GPU, or more memory than your workers should have.
Airflow itself does not make a model reproducible. The training code does, with seeds and a params file. Airflow adds a stable run ID, a clean folder per run, retries and a history of what ran, so that each model can be traced back to one run. The DAG also checks that training uses the same data that was validated.