ETL Pipelines with Apache Airflow
Learn to build Airflow 3 pipelines that survive real failures: DAGs, retries, idempotent loads, backfills, watermarks, CDC concepts, and quality gates.
What You'll Learn
Understanding Why Pipelines Need an Orchestrator
It is 3 AM at acme-shop. A cron job that copies yesterday's orders from Postgres into the warehouse failed four hours ago because the database...
Understanding Airflow 3 Architecture
Most beginner confusion (a task that "should have run" but did not) traces back to not knowing which process does what.
Writing Your First Production DAG
Every DAG file is a Python script that Airflow parses.
Passing Data Between Tasks and Waiting for Other Systems
Tasks need to share small values, and sometimes a pipeline must wait on a file or another DAG.
Making Pipelines Reliable
A pipeline that runs correctly once is easy. One that runs correctly every day, including the days something upstream breaks, is the actual job.
Backfilling and Rerunning Pipelines Safely
Backfilling means running a DAG for a chosen range of past dates, for example when you deploy a new pipeline and want three months of history, or...
Skills You'll Master
Curriculum Index13 topics
Understanding Why Pipelines Need an Orchestrator
It is 3 AM at acme-shop. A cron job that copies yesterday's orders from Postgres into the warehouse failed four hours...
Understanding Airflow 3 Architecture
Most beginner confusion (a task that "should have run" but did not) traces back to not knowing which process does what.
Writing Your First Production DAG
Every DAG file is a Python script that Airflow parses.
Passing Data Between Tasks and Waiting for Other Systems
Tasks need to share small values, and sometimes a pipeline must wait on a file or another DAG.
Making Pipelines Reliable
A pipeline that runs correctly once is easy.
Backfilling and Rerunning Pipelines Safely
Backfilling means running a DAG for a chosen range of past dates, for example when you deploy a new pipeline and want...
Choosing an Ingestion Pattern
Everything so far is about orchestrating a pipeline.
Ingesting from APIs and Files Safely
APIs and file drops are the other two common sources.
Adding Data Quality Gates
Everything up to now ensures a pipeline runs reliably. None of it guarantees the data it moves is correct.
Controlling Concurrency and Observing Pipelines
Airflow runs many tasks in parallel without extra configuration, which is exactly the problem the first time fifty...
Troubleshooting Real Airflow Failures
Debugging Airflow is a skill separate from knowing the concepts.
Hands-On Lab: Building a Reliable Pipeline on Airflow 3
📌 Remember: this lab is free. It needs Docker with Docker Compose and about 6 GB of memory available to containers...
Quick Reference and Common Mistakes
Quick reference Common mistakes Deploying a DAG without writing catchup=False explicitly lets a DAG on an older...
Career Impact
Roles that use the skills in this module.
Data Engineer
Platform Engineer
DevOps Engineer
Release Engineer
Next Modules
Related Guides
Practice on the Coding Sheet
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetFrequently Asked Questions
Cron starts a script on a schedule and knows nothing else. Airflow adds retries, task dependencies, run history, alerting, and backfills. Once a pipeline has more than one step, or someone besides you needs to know it ran, Airflow pays for itself.
No. Airflow schedules and monitors your code. The extraction, transformation, and loading logic still lives in your Python functions, SQL, or Spark jobs, and Airflow tells them when to run and what to do when they fail.
Running a task twice for the same date leaves the data in the same state as running it once. It is what makes retries, reruns, and backfills safe instead of a source of duplicate rows.
Airflow 2 has reached end of life, and Airflow 3 removed or moved several things such as schedule_interval and execution_date. You will still meet Airflow 2 code in older repositories, so this module points out each difference.