Before anything else in this module: you do not need to master data governance to be a working data engineer. If you have built reliable pipelines, modeled data well, and validated data quality, you are already doing the job. Governance is the layer that gets built on top of that reliability, once it exists - it is not a prerequisite for calling yourself competent. That distinction matters because governance content can otherwise feel intimidating and bureaucratic to a learner who is still building their first few pipelines. This module is here because a strong data engineer eventually needs it, not because you needed it yesterday. * A new engineer joins a 200-person company and needs to find the "source of truth" orders table. There are eleven tables with "orders" in the name across three warehouses. Nobody remembers which one is authoritative anymore. That confusion, multiplied across every table in the company, is the exact problem governance exists to prevent. * Governance is fundamentally about four questions asked of every dataset - who owns this, who can see it, how long do we keep it, and where did it come from. Every technique in this module is really just infrastructure for answering those four questions at scale. * Data engineers are not usually the sole owners of governance policy - decisions about what counts as sensitive, how long something must legally be kept, and who is allowed to see what are typically shared across data engineering, security, privacy, legal, and the individual data owners. What a data engineer is responsible for is *implementing* those requirements correctly in the systems they build - the catalog entries, the access policies, the masking, the deletion pipelines. > 📌 **Remember:** governance done well is nearly invisible - a new engineer finds the right table in thirty seconds, a customer's deletion request completes cleanly, an auditor's question gets answered from documentation instead of a scramble through Slack history. Governance done badly is invisible too, until the day it very much is not.
Before any access policy, mask, or retention rule can be applied, you first need to know what kind of data a table actually contains. **Data classification** is the practice of labeling every dataset by sensitivity, so governance decisions can be made consistently instead of case by case. | Classification | Example | |:---|:---| | Public | A public product catalog page | | Internal | Internal operational metrics, not customer-identifying | | Confidential | Revenue figures, business strategy data | | Sensitive / PII | Email, phone number, PAN, bank account details | > **Note:** classification is the first link in a chain, not an isolated checkbox. Once a table is classified `Sensitive / PII`, that single label is what should automatically trigger the rest of this module's practices - it gets an access policy, it gets masked in dev environments, it gets a defined retention period, and it becomes a target for right-to-erasure pipelines. Nothing else in this module can be applied consistently at scale until classification happens first. You will see `PII: yes` or `PII: none` tags throughout the rest of this module - that tag *is* data classification in its simplest form, applied at the table level.
A **data catalog** is a searchable inventory of every data asset in a company - every table, its owner, a description of what it contains, and where to find it. Think of it as the difference between a library with a working card catalog and a library where every book is shelved randomly and you find things only by asking around. ```text Data Catalog Entry Example --------------------------- Table: fct_daily_orders Owner: data-platform-team@company.com Description: One row per restaurant per day. Grain: restaurant_id + order_date. Delivered orders only. Refreshed daily at 6 AM IST by Airflow DAG 'daily_orders_mart'. Tags: PII: none | Domain: orders | Tier: gold Last Updated: 2026-08-15 06:04 IST Downstream: 3 dashboards, 1 ML feature pipeline ``` > **Note:** the single most valuable field in that example is not the description - it is the owner. A table without a clearly documented owner is a table nobody will confidently update, deprecate, or trust six months from now, when everyone who remembers building it has moved teams. ### Catalog approaches - documentation-first, dedicated platforms, or platform-native (Awareness Level) * **Documentation-first** - consistent table and column descriptions inside dbt's own documentation, published with `dbt docs generate`. Zero additional infrastructure, and genuinely enough for most small-to-mid-size teams. * **Dedicated catalog platforms** - open-source options like **Apache Atlas** and **DataHub**, or managed commercial platforms like **Collibra** and **Alation**. More setup and, for the commercial options, real licensing cost, but built specifically for search, discovery, and governance workflows at scale. * **Platform-native governance** - many modern warehouses and lakehouse platforms now ship catalog, tagging, and access-control features directly built in, which can cover a meaningful chunk of this module's needs without adopting a separate tool at all. Check what your specific warehouse or lakehouse platform already offers before assuming a standalone catalog tool is required. > 📌 **Engineering Decision:** for a small-to-mid-size data team just starting to formalise governance, documentation-first is genuinely the right starting point - it costs nothing extra and covers most of what a catalog needs to deliver early on. Reach for a dedicated catalog platform, or lean harder on your warehouse's native governance features, once the number of tables, teams, and warehouses has grown past what dbt docs alone can keep organized and discoverable - usually well into "large, multi-team organization" territory, not on day one. There is no single universal path here; the right choice depends on what your existing stack already gives you for free. > 💡 **Practice:** pick three tables from any pipeline project you have already built in this roadmap and write a catalog entry for each, following the format above - owner, description with explicit grain, tags, and known downstream consumers.
**Data lineage** is the traceable path a piece of data takes from its original source, through every transformation, to wherever it finally lands. When a number on a dashboard looks wrong, lineage is what lets you walk backward and find exactly which upstream table, transformation, or pipeline run introduced the problem. orders (Postgres) | v raw_orders (S3, Bronze) | v stg_orders (dbt staging model) | v fct_daily_orders (dbt mart model) | v Revenue Dashboard (BI tool) > **Note:** this kind of lineage - showing how tables and models depend on one another - is called **dataset-level lineage**, and it is what the diagram above demonstrates. **Column-level lineage** goes a level deeper, tracing an individual field through every transformation it passed through - for example, showing that a `revenue_usd` column was calculated from `amount` and `exchange_rate` specifically. Not every tool can infer column-level lineage automatically, so it is worth knowing the two are different claims, even though dataset-level lineage is usually the one you get for free. > **Note:** within your dbt project, `ref()` and `source()` build this dependency graph automatically, and `dbt docs generate` turns it into a browsable graph - that gives you strong lineage for anything dbt manages. End-to-end lineage across your *entire* stack - a SaaS application, through Kafka, through a custom Python ingestion script, into S3, through Spark, and only then into dbt - is a bigger claim, and it typically requires additional integrations or a broader metadata platform to connect those non-dbt hops, since dbt has no visibility into what happened before data reached its own sources. > 🔴 **Common Mistake:** treating lineage as something you will "add later" once a pipeline is working leads to gaps that are painful to reconstruct after the fact - once three engineers have left the team and the original context is gone, tracing an undocumented pipeline's history can take days instead of the seconds a working lineage graph would have taken. Wire up `ref()` and `source()` correctly in dbt from the very first model, and lineage documents itself as you go.
**Role-based access control (RBAC)** grants data access based on a person's role - "analyst," "finance," "data engineer" - rather than granting or revoking permissions for each individual person one at a time. This scales far better than manually managing access for every new hire and every departing employee. ```sql -- Create a role and grant it access to a specific schema CREATE ROLE analyst_role; GRANT USAGE ON SCHEMA analytics TO analyst_role; GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO analyst_role; -- Assign a real user to that role - access management stays centralised GRANT analyst_role TO "priya.sharma@company.com"; ``` ### Row-level security and column-level access - finer control than a whole table Sometimes access needs to be more precise than "can see this table" or "cannot see this table." **Row-level security** restricts which *rows* a role can see. **Column-level access** is a related but distinct idea worth separating into three layers: * **Column-level access control** - can this role query this column at all, enforced natively by the database's own permission system * **Dynamic data masking** - the role can query the column, but the value returned is transformed or hidden based on who is asking, often a native warehouse feature * **View-based masking** - a secure view exposes different values per role using conditional logic, which is what the SQL example below demonstrates - it is a valid and common pattern, but it is application/query-level masking, not the same mechanism as your warehouse's native column-level security feature. Which mechanism is actually available depends on your specific platform. ```sql -- Enable row-level security on the table first - policies do nothing until this runs ALTER TABLE orders ENABLE ROW LEVEL SECURITY; -- Row-level security example - a regional manager only sees their own city's orders -- The ", true" argument to current_setting makes this return NULL instead of erroring -- if the session variable was never set, which fails safely closed rather than crashing open CREATE POLICY city_manager_policy ON orders FOR SELECT USING ( city = current_setting('app.current_user_city', true) ); -- View-based masking - customer service sees masked phone numbers, finance sees real ones SELECT order_id, CASE WHEN current_role() = 'finance_role' THEN customer_phone ELSE CONCAT('XXXXX', RIGHT(customer_phone, 4)) END AS customer_phone FROM orders; ``` > ⚠️ **Security:** in production, the application layer must set `app.current_user_city` safely for each session or request, based on who is actually logged in - never let an untrusted client set this value directly, or a user could simply claim any city and bypass the policy entirely. > 💡 **Practice:** design an access policy table on paper for a Zerodha-style trading platform with three roles - `support_agent`, `compliance_officer`, and `data_engineer` - listing which tables and which specific columns (for example, PAN number, bank account number) each role should and should not be able to see.
Production data often ends up copied into a development or staging environment so engineers can test against realistic data - and that is exactly how a customer's real phone number, PAN number, or address ends up sitting unprotected on a junior engineer's laptop. **Data masking** replaces sensitive real values with realistic but fake ones before that data reaches a non-production environment. ```sql -- Masking customer PII when copying production data into a dev environment SELECT order_id, customer_id, CONCAT('customer_', customer_id, '@masked.dev') AS email, -- deterministic fake email CONCAT('9', LPAD((customer_id % 900000000)::text, 9, '0')) AS phone, -- fake but valid-shaped amount, order_date FROM orders; ``` > **Note:** notice the masked email and phone are *deterministic* - the same `customer_id` always produces the same fake email. This principle matters more than this one example: deterministic masking is useful whenever the same source value must always map to the same masked value across multiple datasets, so that joins and referential relationships between tables keep working correctly on masked data - a developer testing a join between `orders` and `customers` needs the masked values to still match consistently across both tables, or the test data breaks in ways that have nothing to do with the actual bug being investigated. > 🔴 **Common Mistake:** treating "masking," "pseudonymisation," and "anonymisation" as interchangeable terms is a common and costly imprecision. **Masking** hides a value from a particular viewer, often reversibly, and is what the dev-environment example above does. **Pseudonymisation** replaces an identifier with a token or hash, but re-identification may still be possible if someone has access to the mapping or enough auxiliary data - a hashed email is still personal data under most privacy frameworks. **Anonymisation** is a stronger standard: the goal is that the person can no longer reasonably be re-identified at all, even by combining the dataset with other available information. > 📌 **Remember:** avoid putting real production PII into non-production environments whenever you genuinely can. In rough order of preference: synthetic data generated to look statistically realistic is the safest option where it is workable; anonymised data is appropriate for lower-risk analytical use once re-identification is no longer reasonably possible; deterministic masking or pseudonymisation is useful when a team genuinely needs realistic, joinable, production-like relationships preserved for testing; and strict access controls are what you fall back on for the cases where sensitive data cannot be avoided in a given environment at all. "The team is trusted" is not by itself a reason to skip these layers - trust does not prevent an accidental leak, a misconfigured export, or a laptop left in a cab. Separately, masking a customer's name and email but leaving a unique identifier like PAN number or Aadhaar number untouched still allows the record to be re-identified, since a unique government ID is functionally as identifying as a name - whichever of the three techniques above you are applying, apply it to every field that could uniquely trace back to a person, not just the obviously personal-looking ones. ### Data minimisation and retention policies - deciding how long data lives before deciding how it dies * **Data minimisation** means only collecting and storing the fields actually needed for a defined business purpose - not pulling every field an API happens to offer just because it is there. * **Retention policies** define how long data is kept before automatic deletion - raw clickstream logs might be kept 90 days, while financial transaction records might legally require 7 years, and a pipeline's S3 lifecycle policies should reflect exactly these differences per data type, not one blanket setting for everything. Retention comes before right-to-erasure in this module deliberately - the more disciplined a team is about not keeping data longer than it needs to in the first place, the smaller and simpler the right-to-erasure problem becomes, since there is simply less data scattered across fewer places by the time a deletion request ever arrives.
Before anything else in this module: you do not need to master data governance to be a working data engineer. If you hav...
Before any access policy, mask, or retention rule can be applied, you first need to know what kind of data a table actua...
A data catalog is a searchable inventory of every data asset in a company - every table, its owner, a description of wha...
Data lineage is the traceable path a piece of data takes from its original source, through every transformation, to wher...
Role-based access control (RBAC) grants data access based on a person's role - "analyst," "finance," "data engineer" - r...
Production data often ends up copied into a development or staging environment so engineers can test against realistic d...
GDPR (General Data Protection Regulation) is a European data protection law, but its practical patterns have become the ...
A Flipkart-style company runs a right-to-erasure pipeline that successfully deletes a customer from the goldcustomerorde...
Federated governance is the data mesh approach to this whole module's problem, at a much larger organizational scale - i...
Every technique in this module fits into one repeating cycle. New data enters the company, and this loop is what keeps i...
Build a data catalog entry, in the format shown earlier in this module, for every table produced in your Required Projec...
Task Tool / Pattern Label a dataset's sensitivity Data classification (Public/Internal/Confidential/PII) Document a tabl...
Treating a data catalog as a one-time documentation project instead of a living practice means it is accurate on launch ...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.