The Airflow DAG for Flipkart's daily orders pipeline finished at 4:03 AM. Every task shows green. The dashboard says SUCCESS. Nobody gets paged. But that morning, 15% of `customer_id` values in the orders table are null. A handful of `payment_status` values now say `"cancled"` instead of `"cancelled"`, because a source system shipped a typo three days ago. Revenue for a few hundred rows is negative because of a refund-handling bug upstream. None of this crashed anything. The pipeline ran exactly as designed - it extracted, transformed, and loaded data, and every one of those steps technically succeeded. By 9 AM, a merchandising analyst is building a report on top of that data. By 10 AM, someone in a leadership meeting is looking at a revenue number that is quietly wrong. > 📌 **Remember:** A successful pipeline run means the code executed without > crashing. It says nothing about whether the data it produced is actually > correct. These are two different guarantees, and most pipelines only check > the first one. Think of it like airport security. A flight can take off exactly on schedule, with a fully fueled plane and a qualified crew - and none of that tells you whether anything dangerous made it through the gate. Operational success and safety are checked separately, on purpose, because one does not imply the other. Data quality checks are your data's security checkpoint - a deliberate, separate gate that data passes through before you trust it downstream. As covered in the Airflow module, an orchestrator tracks whether tasks ran and whether they threw an error. It has no idea whether the CSV your API pulled in today has a null rate that jumped from 2% to 40%. That is a different kind of failure, and it needs a different kind of tool. ### What this module builds toward By the end of this module you will be able to define **expectation suites** that codify what "correct" means for a dataset, wire them into a pipeline as **checkpoints**, and make deliberate decisions about which failures should stop a pipeline versus which should just raise a warning. > **Note:** This module is written against the Great Expectations **1.x > fluent API** (`gx.get_context()`, `context.suites`, `context.checkpoints`, > and so on) - the modern, code-first approach. If you find an older > tutorial online using `BatchRequest` objects or a `great_expectations.yml` > CLI scaffold, that syntax belongs to an earlier major version and is not > what current documentation recommends. Minor releases within 1.x can still > shift small details, so pin the exact version you install (`pip install > "great_expectations==1.3.*"` or similar) for your project and check the > official docs if a method name has moved by the time you're reading this. ### A quick self-check before moving on * In your own words, what is the difference between "the pipeline succeeded" and "the data is correct"? * Why doesn't Airflow's task-success status catch a null-rate spike? ---
"Bad data" is not one problem - it is at least four different questions, and each one needs a different kind of check. Treating them as one blob is how teams end up with a pile of ad-hoc `if` statements that don't actually cover the failure modes that matter. | Dimension | Question it answers | Example failure | Example check | |:---|:---|:---|:---| | Accuracy | Is the value actually correct? | PhonePe transaction amount does not match the source ledger | Reconcile source vs target sums within a tolerance | | Completeness | Is anything missing? | Swiggy order row has a null `restaurant_id` | `expect_column_values_to_not_be_null` | | Consistency | Does it agree with itself and its rules? | Flipkart order status uses `"cancled"` in some rows, `"cancelled"` in others | `expect_column_values_to_be_in_set` | | Timeliness | Did it arrive when expected? | Yesterday's sales data lands 8 hours late | Freshness check on max `event_time` vs now | > **Note:** These four are not academic categories to memorize for their own > sake - they are a diagnostic tool. When you're deciding what to check on a > new dataset, running through accuracy, completeness, consistency, and > timeliness in order is a fast way to make sure you haven't left an obvious > gap. ### Concept check * A payments table has zero nulls in `amount`, but 3% of rows have the same `transaction_id` appearing twice. Which dimension does that failure belong to? * A dataset lands in S3 on time, has no nulls, and has no duplicates - but `amount` is stored as the wrong currency for 500 rows. Which dimension is that? ---
Before looking at any code, here's the full shape of how these pieces fit together - source, validation, severity decision, and destination: Source | v Raw Validation | +------------------+ | | Critical Non-critical | | STOP Quarantine / Warning | | v v Alert Transform | v Transformed Validation | v Trusted Data Keep this picture in mind through the rest of the module - every section below fills in one piece of it: how validation actually runs (Great Expectations Fundamentals), what to check for (Writing Expectations), how severity gets decided (Validation Result vs Enforcement Policy), and how it plugs into a real DAG (Great Expectations Inside an Airflow DAG). ---
Validation only earns its keep if it happens at the right point in a pipeline - after data lands, before anything downstream trusts it. Raw source | v Validation | PASS / FAIL / \ v v Transform Quarantine | v Warehouse / Lakehouse Not every failed check deserves the same response. A `transaction_id` that suddenly allows nulls is a **critical** failure - stop the pipeline, because downstream joins and financial reconciliation will silently produce garbage if this data proceeds. A slight day-over-day increase in null `promo_code` values might be a **warning** - worth a Slack message, not worth blocking today's revenue numbers from loading. > 📌 **Engineering Decision:** Fail the pipeline immediately vs quarantine bad > records. Fail immediately when the violation would corrupt something > downstream that many other things depend on - a broken primary key, a > schema that no longer matches what every downstream model assumes. Quarantine > when the violation affects only a subset of rows and the rest of the batch > is still valuable - route the bad 200 rows to a `quarantine` table or > prefix, load the other 1.8 million, and alert someone to fix the source. > Blocking an entire day's pipeline because 0.01% of rows are malformed is > usually the wrong trade - but blocking it because your primary key is no > longer unique usually is the right one. > 💡 **Practice:** For each of these, decide whether it should block the > pipeline or just quarantine the affected rows and continue: (1) > `transaction_id` is no longer unique, (2) 12 out of 2 million rows have a > null `city`, (3) `payment_status` contains a value that isn't in your > known set of statuses. ---
Before any syntax, the mental model: Dataset | v Expectations (the rules you define) | v Validation (running the rules against real data) | +--------+ | | PASS FAIL | | v v Continue Stop / Alert / Quarantine **Great Expectations** (GX) is a Python library purpose-built for exactly this loop. Instead of writing scattered `assert` statements or manual `df.isnull().sum()` checks buried inside a pipeline script, GX gives you a structured, reusable way to declare rules and get a clear pass/fail result you can act on. The core vocabulary, in the order you'll actually use it: * **Expectation** - a single, specific rule about your data, like "this column must never be null" or "this column's values must be positive." * **Expectation Suite** - a named, reusable collection of expectations for one dataset, stored so it can be version-controlled like code. * **Validation** - the act of running a suite against real data and getting a result back. * **Checkpoint** - the object that ties a suite to a specific batch of data and runs the validation, producing a `success` flag your pipeline can branch on. * **Data Docs** - an auto-generated HTML report showing what passed, what failed, and why, for every validation run. ```python import great_expectations as gx ## get_context() is the entry point to everything GX does in this session context = gx.get_context() ## point GX at a pandas DataFrame you already loaded - GX also supports ## Postgres, Snowflake, Spark, and other backends the same way data_source = context.data_sources.add_pandas(name="payments_source") data_asset = data_source.add_dataframe_asset(name="payments_df") batch_definition = data_asset.add_batch_definition_whole_dataframe("payments_batch") ``` > **Note:** `context` is your session's connection to GX - it tracks every > suite, data source, and checkpoint you define. You typically create it > once per script or notebook and reuse it throughout. ---
Before writing a single expectation, it helps to ask the same seven questions for any new dataset. This is more durable knowledge than memorizing GX's expectation classes - it's the checklist you'll run through mentally for every table you're ever handed, in any tool: 1. **Schema** - are the columns you expect actually present? 2. **Completeness** - which fields must never be null? 3. **Uniqueness** - what identifies one record, and can it repeat? 4. **Validity** - what ranges or allowed values are legitimate? 5. **Relationships** - does this data correctly reference other entities? 6. **Volume** - is today's row count plausible compared to history? 7. **Business invariants** - what combinations of values should never occur? The six GX expectation categories below map directly onto the first four of these questions. Relationships and volume get their own treatment shortly after. Business invariants - the most valuable and least generic category - get a full section of their own later in this module.
The Airflow DAG for Flipkart's daily orders pipeline finished at 4:03 AM. Every task shows green. The dashboard says SUC...
"Bad data" is not one problem - it is at least four different questions, and each one needs a different kind of check. T...
Before looking at any code, here's the full shape of how these pieces fit together - source, validation, severity decisi...
Validation only earns its keep if it happens at the right point in a pipeline - after data lands, before anything downst...
Before any syntax, the mental model: Dataset v Expectations (the rules you define) v Validation (running the rules again...
Before writing a single expectation, it helps to ask the same seven questions for any new dataset. This is more durable ...
Real data quality work leans on a handful of recurring patterns, not an exhaustive catalog of every possible rule. Learn...
Picture the alternative: a pipeline script sprinkled with if row['amount'] <= 0: raise ValueError(...) in five different...
An expectation suite by itself doesn't do anything until you run it against real data. That's the checkpoint's job - it ...
It's tempting to think an expectation like ExpectColumnValuesToBeInSet somehow "knows" whether its failure should be cri...
As covered in the Airflow module, a DAG is a graph of tasks with dependencies. Data quality checks are just another task...
You already know dbt tests from the Analytics Engineering module - notnull, unique, acceptedvalues, relationships, runni...
The generic checks so far - not null, unique, in a set - catch a lot. The checks that actually prevent real incidents us...
A Flipkart-style daily orders pipeline normally processes 2 million rows. This morning: Only 200,000 rows arrived. trans...
A single pass or fail is a snapshot. The more useful signal is the trend - watching whether validation health is degradi...
Python - the custom validation logic and SQL-backed expectations you write here are ordinary Python, using the same skil...
Prerequisites: Python environment from the Python module, pip install greatexpectations, and a sample CSV of payment tra...
Term / Concept What it means Expectation A single rule about a dataset Expectation Suite A named, reusable, version-cont...
Retrying a failed data validation task multiple times even though the source file genuinely contains invalid values. Ret...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.