Data Quality with Great Expectations
Learn to stop bad data before anyone trusts it: Great Expectations suites, checkpoints, severity rules, quarantine, data contracts, and Airflow gates.
What You'll Learn
Understanding Why a Successful Pipeline Can Still Fail You
You have joined acme-shop's data team, and the orders pipeline you will protect has run for months.
Understanding Great Expectations Fundamentals
Great Expectations (GX) is a Python library for declaring rules about data and getting a clear pass or fail result you can act on.
Writing Expectations for the Patterns You Will Use
Before writing any expectation, ask the same seven questions of every new dataset.
Designing Data Contracts
A data contract is a written agreement between the team that produces a dataset and the teams that consume it.
Running Checkpoints and Reading Results
An expectation suite does nothing until you run it against real data.
Applying a Severity Policy and Quarantine
An expectation does not know whether its failure is critical. GX tells you whether a rule passed. What your pipeline does next is your decision.
Skills You'll Master
Curriculum Index13 topics
Understanding Why a Successful Pipeline Can Still Fail You
You have joined acme-shop's data team, and the orders pipeline you will protect has run for months.
Understanding Great Expectations Fundamentals
Great Expectations (GX) is a Python library for declaring rules about data and getting a clear pass or fail result you...
Writing Expectations for the Patterns You Will Use
Before writing any expectation, ask the same seven questions of every new dataset.
Designing Data Contracts
A data contract is a written agreement between the team that produces a dataset and the teams that consume it.
Running Checkpoints and Reading Results
An expectation suite does nothing until you run it against real data.
Applying a Severity Policy and Quarantine
An expectation does not know whether its failure is critical. GX tells you whether a rule passed.
Writing Business Rules with SQL Expectations
The checks that prevent real incidents usually come from someone who understands the business, not from a rule library.
Putting Quality Gates in an Airflow 3 DAG
Data quality checks are just another task in the DAG. Their job is to gate what runs next.
Choosing Between Great Expectations and dbt Tests
You will meet dbt tests in the Analytics Engineering with dbt module: not_null, unique, accepted_values, and...
Troubleshooting the Pipeline That Said SUCCESS
This morning's acme-shop orders run is green.
Running the Hands-On Lab
You will validate acme-shop's dirty dataset, watch a critical failure block the run, quarantine what you can, and prove...
Reviewing Quick Reference and Common Mistakes
Quick reference Common mistakes Retrying a failed validation task several times.
Understanding What You Built and What Comes Next
What did you build? You wrote a data contract for acme-shop orders and generated an expectation suite from it.
Career Impact
Roles that use the skills in this module.
Data Engineer
Platform Engineer
Next Modules
Practice on the Coding Sheet
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetFrequently Asked Questions
An expectation is one rule, such as 'order_id must be unique'. A suite groups the rules for one dataset. A checkpoint runs a suite against real data and returns a pass or fail result your pipeline can act on.
No. Great Expectations only reports which rules passed and failed. Your pipeline code decides whether a failure should block the run, raise a warning, or quarantine rows.
Use Great Expectations for data that is not in the warehouse yet, such as raw files, API responses, and DataFrames. Use dbt tests for models already in the warehouse. Most teams use both.
A DataFrame batch needs the data at run time. Pass it with checkpoint.run(batch_parameters={"dataframe": df}). The suite and checkpoint are saved, but the data itself never is.