This module is deliberately AWS-focused - it teaches AWS's data services in depth rather than spreading thin across AWS, GCP, and Azure equivalents. If your team runs on a different cloud, the concepts (data lake zoning, partitioning, catalog-driven query engines, pipeline IAM) transfer directly, but the exact service names will differ - the short GCP-equivalents section later in this module is a pointer, not a parallel deep dive. If you have not yet worked through EC2, VPC, and IAM fundamentals, go do that first in the **Cloud Engineering Mastery** roadmap - specifically its Cloud Computing and AWS Fundamentals and AWS Core Services steps. That roadmap is where you learn what a VPC actually is, how IAM policies work at the concept level, and how EC2 instances get provisioned. It is the deep, general-purpose cloud hub, and duplicating it here would waste your time re-reading things you already know. This module assumes that foundation and goes somewhere different: the handful of AWS services a data engineer touches every single day, used the way a data engineer actually uses them. That is a narrower and more specific slice than "AWS" as a whole. > 📌 **Remember:** this module is the data-specific layer, not a second cloud fundamentals course. Every service below gets used constantly in real data pipelines - S3 as a lake, Glue as a catalog and ETL engine, Athena for ad-hoc SQL, Redshift as a warehouse, IAM roles wired specifically for pipeline authentication. An analyst at a Flipkart-scale company does not think about EC2 instance types or VPC route tables day to day. They think about which S3 prefix has today's orders, whether the Glue Crawler picked up the new column, and why an Athena query just scanned 900 GB instead of 9 GB. That is the world this module lives in. ### The architecture this module builds, end to end Every service below fits into one continuous pipeline. Keep this picture in mind as each piece gets introduced individually. Sources | v +---------------+ | S3 | | Raw Zone | +-------+-------+ | v +---------------+ | Glue / Spark | | Transformation| +-------+-------+ | v +---------------+ | S3 Processed | | Parquet | +-------+-------+ | +---------+---------+ | | v v Athena Redshift | | +---------+---------+ | v BI S3 stores every byte at every stage. Glue Catalog holds the metadata that lets Athena and Redshift Spectrum find and understand that data. Glue ETL jobs do the actual transformation work between zones. Athena and Redshift are two different ways of querying the result, chosen based on the Engineering Decision covered later in this module. IAM roles authorize every arrow in this diagram, and Terraform provisions every box.
An S3 bucket with a thousand CSV files thrown into one flat folder is not a data lake. It is a junk drawer with a price tag. The difference between the two is structure, and structure is the entire skill. ### The three-zone pattern s3://swiggy-data-lake/ raw/ <- landing zone, exactly as received orders/ processed/ <- cleaned, typed, deduplicated orders/ curated/ <- business-ready, aggregated daily_revenue/ * **raw/** stores source data in its original format, untouched, acting as an immutable audit trail - if a downstream transformation has a bug, you can always replay from raw * **processed/** holds cleaned, standardized data, typically in Parquet, with nulls handled and duplicates removed * **curated/** holds highly aggregated, business-ready tables that BI tools and dashboards query directly > 📌 **Engineering Decision:** never let a transformation job write directly into `raw/`. Treat `raw/` as append-only and immutable by convention - the moment something in `processed/` or `curated/` turns out wrong, you need to be able to replay it from an unmodified source. S3 itself does not enforce this automatically; it is an architectural policy your team agrees on and enforces through IAM permissions (write access to `raw/` restricted to the ingestion role only) and, where the source system demands stronger guarantees, S3 Object Lock or bucket versioning. If a bug corrupts `raw/` itself with no such enforcement in place, you have lost your ability to recover. ### Partitioning by date - the single highest-leverage decision you will make **Partitioning** means organizing files into a folder structure that encodes a column's value directly in the path, so a query engine can skip reading folders that do not match its filter. s3://swiggy-data-lake/processed/orders/ year=2026/month=08/day=14/ part-0001.parquet year=2026/month=08/day=15/ part-0001.parquet ```sql -- Athena reads ONLY the day=15 folder because the query filters on it select restaurant_id, sum(order_amount) from orders where year = 2026 and month = 8 and day = 15 group by restaurant_id; ``` > **Note:** this is called Hive-style partitioning - the folder name itself contains `column=value`, which lets Athena and the Glue Catalog match query filters to specific folders without scanning anything else. Without it, that same query scans every file in the entire `orders/` prefix, every single time. An unpartitioned table forces every query to scan every file, no matter how narrow the actual `WHERE` clause is, because Athena charges by bytes scanned rather than rows returned. If partition pruning reduces a query from scanning 1 TB down to 10 GB, the scanned-data cost for that query is roughly 100 times smaller - not because partitioning inherently produces a 100x discount, but because that specific query now reads a specific fraction of the table. The actual savings on any given query depend entirely on how selective your filter is relative to the total dataset. > 🔴 **Common Mistake:** partitioning by a high-cardinality column like `user_id` or `order_id` instead of a low-cardinality one like `date` or `region` creates millions of tiny partitions, each holding a few kilobytes of data. This makes metadata lookups slower than the scan itself was supposed to save. Partition on the columns your queries actually filter on, and keep cardinality low - date, region, or category, not unique IDs. > 🔴 **Common Mistake:** technically correct partitioning can still produce a terrible data lake if each partition ends up holding thousands of tiny files instead of a few well-sized ones. Partition pruning reduces how much data gets selected, but a huge count of small files adds real metadata lookup and job-scheduling overhead on top of that. Production pipelines periodically compact small files within a partition into fewer, appropriately sized objects - this matters as much as the partitioning scheme itself. ### File format selection - why Parquet wins almost every time **Parquet** is a columnar file format, meaning it stores all values from one column together on disk, instead of storing a full row together like CSV does. Analytical queries usually read a handful of columns out of dozens - columnar storage lets the engine skip every column it doesn't need entirely. | Format | Storage Layout | Best For | |:---|:---|:---| | CSV | Row-based, plain text | Small files, human inspection, initial raw drops | | JSON | Row-based, nested | Semi-structured API responses in raw zone | | Parquet | Column-based, compressed | Everything in processed/ and curated/ zones | > **Note:** Parquet is a file format, not a full table-management system - it has no built-in concept of transactions, schema evolution across files, or snapshots. Modern lakehouse table formats such as Apache Iceberg and Delta Lake add those capabilities on top of Parquet files sitting in S3. This module sticks to plain Parquet; table formats get proper coverage in the Modern Data Warehousing and the Lakehouse module later in this roadmap. > 🔴 **Common Mistake:** using `SELECT *` in Athena against a wide Parquet table reads every column instead of only the ones the query actually needs, even when partition pruning is working perfectly. Partition pruning controls how many files get scanned; column selection controls how much of each file gets read. Both matter - select only the columns your query needs, especially on tables with dozens of columns. Converting raw CSV to Parquet in the processed zone can substantially reduce Athena bytes scanned through column pruning and compression, but the actual reduction depends heavily on the dataset - column count, data types, how many columns your queries actually select, and how well the data compresses. Do not treat any specific percentage as a guarantee; measure the scanned-bytes difference on your own data the way the hands-on lab below has you do. > 💡 **Practice:** take a CSV file of Zerodha-style trade data, load it into a pandas DataFrame, write it out as both CSV and Parquet, and compare the two file sizes on disk. Then write an Athena-style query in your head - which columns would it actually need to read - and reason through why Parquet would scan less data for that same query. ### Lifecycle policies - stopping storage costs from growing forever A **lifecycle policy** is an S3 rule that automatically moves or deletes objects based on their age, without anyone running a manual cleanup job. ```json { "Rules": [ { "ID": "archive-old-raw-orders", "Filter": {"Prefix": "raw/orders/"}, "Status": "Enabled", "Transitions": [ {"Days": 90, "StorageClass": "GLACIER_IR"} ] } ] } ``` > **Note:** AWS offers several Glacier-family storage classes with different retrieval speeds and costs - Glacier Instant Retrieval (`GLACIER_IR`, used above, milliseconds to access), Glacier Flexible Retrieval (minutes to hours), and Glacier Deep Archive (up to 12 hours, cheapest). Pick based on how quickly you would realistically need that 90-day-old data back if a replay were ever needed - the conceptual point is the same across all three: cheap, slow-retrieval storage for data you rarely query but must legally or operationally retain. ### S3 event notifications - triggering work the moment a file lands An **S3 event notification** fires automatically on specific event types - most commonly object creation or deletion - and can invoke a Lambda function directly, no polling, no scheduled job checking "did a new file arrive yet." > **Note:** for simple workflows, S3 invoking Lambda directly, as shown below, is fine. For more resilient production pipelines, route S3 events through SQS or EventBridge instead, so a consumer can retry on failure, buffer bursts of incoming files, and process events independently of Lambda's own concurrency limits. Direct S3-to-Lambda is the simplest starting point, not automatically the production pattern. ```python import boto3 from urllib.parse import unquote_plus s3 = boto3.client('s3') def validate_new_file(event, context): """ Triggered by an S3 ObjectCreated event. Validates that a newly landed CSV has the required columns before it is allowed into the processed zone. """ # The event tells us exactly which file triggered this - no need to list the bucket bucket = event['Records'][0]['s3']['bucket']['name'] # S3 keys with spaces or special characters arrive URL-encoded - decode before using key = unquote_plus(event['Records'][0]['s3']['object']['key']) required_columns = {'order_id', 'restaurant_id', 'amount', 'status'} header = _read_csv_header(bucket, key) if required_columns.issubset(header): # valid - promote toward processing, downstream Glue job picks it up _move_object(bucket, key, key.replace('raw/', 'validated/')) else: # invalid - isolate it, never let it silently flow downstream missing = required_columns - set(header) _move_object(bucket, key, key.replace('raw/', 'quarantine/')) _send_alert(f"Schema check failed for {key}: missing {missing}") ``` > **Note:** this is the validation -> quarantine -> alert pattern, and it is worth naming explicitly because it applies everywhere in data engineering, not just here: an incoming file passes or fails a schema check, valid files continue toward processing, invalid files get isolated in a `quarantine/` prefix instead of silently reaching downstream consumers, and someone gets alerted either way. Bad data should fail loudly and get isolated - never pushed downstream quietly hoping nobody notices. Incoming file | v Schema validation / \ valid invalid | | v v validated/ quarantine/ | | v v process alert
**AWS Glue** is actually a broader service family, but for the purposes of this module, think of it primarily as two separate tools: a **Data Catalog** that stores table metadata, and a serverless **ETL engine** built on Apache Spark that transforms data between zones. These are the two capabilities a data engineer reaches for daily. You can use the Glue Data Catalog without ever running a Glue ETL job. Glue ETL jobs can also read directly from supported sources without relying on the Catalog, although many production workflows do use it for schema and table metadata rather than hardcoding schema details into the job itself. ### The Glue Data Catalog - one metastore, shared everywhere The Glue Data Catalog is a Hive-compatible metastore. It stores database and table definitions - column names, types, partition keys, S3 locations - that Athena, Redshift Spectrum, and EMR all read from the same source of truth. Define a table's schema once in Glue, and every query engine on the platform sees it identically. ### Glue Crawlers - automatic schema discovery A **Glue Crawler** scans data sitting in S3, infers the schema, and registers or updates the corresponding table in the Data Catalog. ```bash ## Run a crawler against the processed orders prefix aws glue start-crawler --name swiggy-orders-crawler ``` Expected output: ```text { "CrawlerName": "swiggy-orders-crawler" } ``` > 🔴 **Common Mistake:** running a Crawler on every single pipeline execution, even when the schema never changes, wastes both time and money - Crawlers charge by the hour they run. Run a Crawler once to establish the schema, and only re-run it when you expect an actual schema change, not on every ingestion cycle. ### Schema evolution - when tomorrow's file looks different A data lake does not guarantee that every file has the same schema forever. Producers change APIs, add columns, rename fields, and occasionally change data types - and none of that stops the pipeline from receiving a file. Monday's file: order_id | customer_id | amount Tuesday's file: order_id | customer_id | amount | coupon_code Adding a new nullable column, like `coupon_code` above, is usually the easiest kind of change to absorb - a re-run Crawler picks it up and existing queries that do not reference the new column keep working. Renaming a column or changing an existing column's data type (say, `amount` going from a number to a string) is far more dangerous, because every downstream model that reads `amount` as a number now silently breaks or produces wrong results. > 📌 **Engineering Decision:** treat a source schema change as a pipeline change, not a harmless file change. Validate the incoming schema before promoting new data into the processed zone, and re-test downstream models whenever a source schema changes - the validation Lambda from the S3 event section earlier is exactly the right place to catch this before bad data spreads further downstream. ### Glue ETL jobs - transformation without managing a cluster A Glue ETL job runs a PySpark script on a fully managed, serverless Spark cluster you never provision yourself. You pay for **DPU** (Data Processing Unit) time only while the job runs. > 📌 **Engineering Decision:** start with the smallest practical worker configuration for a Glue job, measure actual runtime and resource utilization, then scale based on that observed behavior rather than picking a fixed DPU count upfront - worker types, Glue versions, and workload shuffle characteristics all affect what "right-sized" actually means for a given job. Job bookmarks can help Glue track previously processed data and reduce unnecessary reprocessing for supported sources and job patterns, but they are not a universal incremental-processing guarantee - validate bookmark behavior against your specific source and transformation logic rather than assuming it "just works" for every pipeline.
**Amazon Athena** lets you run standard SQL against data sitting in S3, using the Glue Data Catalog for schema, with no cluster to provision or manage. You pay based primarily on the amount of data scanned, subject to AWS's current pricing - check the AWS pricing page for the exact rate rather than relying on a fixed number, since pricing and features like result reuse change over time. ```sql -- Reads only the year=2026/month=08/day=15 partition because of the WHERE clause select restaurant_id, count(*) as order_count, sum(order_amount) as total_revenue from orders where year = 2026 and month = 8 and day = 15 and order_status = 'delivered' group by restaurant_id order by total_revenue desc limit 10; ``` > **Note:** Athena is genuinely serverless in the sense that matters to a data engineer - there is no cluster sizing decision to make, no idle compute to pay for between queries. The tradeoff is that every single query, however small, pays its own per-byte-scanned cost, which is why partitioning and Parquet matter so much here specifically. > **Note:** this is a common point of beginner confusion, worth pinning down explicitly. S3 stores the actual data files. The Glue Catalog stores metadata *about* that data - column names, types, partition keys, and the S3 location - but holds none of the data itself. Athena executes SQL by reading the Catalog to know what to look for, then reading the actual bytes from S3. Three distinct jobs, one pipeline: `S3 (data) -> Glue Catalog (metadata) -> Athena (query engine)`.
**Amazon Redshift** is AWS's analytical data warehouse, available both as provisioned clusters (you manage cluster size, pay by the hour whether actively querying or not) and as a serverless deployment (compute scales automatically, billed by usage). This module focuses on the warehouse workload pattern rather than the deployment-model decision - the Engineering Decision below assumes the traditional provisioned-versus-Athena framing, which is still the most common comparison you will encounter, but know that Redshift Serverless exists as a middle option worth checking when the provisioned-vs-Athena tradeoff is close. > 📌 **Engineering Decision:** choose Athena when queries are occasional or unpredictable - you only pay when you actually query. Choose Redshift when the same complex queries run repeatedly, often, by many users - a fixed hourly cluster cost becomes cheaper than per-query billing once query volume crosses a certain threshold, and Redshift's ability to keep hot data in cluster storage makes repeated complex joins faster than re-scanning S3 every time. There is no universal crossover point - it depends on query frequency and complexity, so when the choice is close, run the cost comparison for your actual workload rather than trusting a rule of thumb. > **Note:** Athena and Redshift are not mutually exclusive in practice. Many companies keep large historical datasets sitting in S3, query them occasionally through Athena, and load only the high-value curated subset into Redshift for fast, frequent BI workloads. The decision above is about which engine handles a given query pattern, not about picking one tool for the entire company. > 💡 **Tip:** once data is loaded into Redshift, table design decisions like distribution style and sort keys can significantly affect join and query performance - those warehouse-specific optimization techniques get proper coverage in the Modern Data Warehousing and the Lakehouse module later in this roadmap. For now, know that loading data into Redshift is not the end of the performance story.
**Kinesis Data Streams** ingests continuous, real-time event data - think a live feed of PhonePe transaction events - that consumers read from in near real time. **Kinesis Firehose** is the simpler sibling - it delivers streaming data straight into S3, Redshift, or OpenSearch with minimal configuration, when you do not need custom stream processing logic. > 💡 **Tip:** if your requirement is simply reliable delivery into S3, Redshift, or another supported destination with minimal stream-processing logic, Firehose is often the simpler choice. Reach for Data Streams instead when applications need direct control over consumption, replay from a specific point, multiple independent consumers, or low-latency custom stream processing - Firehose does not give you those controls. This module only introduces Kinesis at a glance - full depth on streaming architecture is covered later in this roadmap, in the Streaming Data Engineering module.
This module is deliberately AWS-focused - it teaches AWS's data services in depth rather than spreading thin across AWS,...
An S3 bucket with a thousand CSV files thrown into one flat folder is not a data lake. It is a junk drawer with a price ...
AWS Glue is actually a broader service family, but for the purposes of this module, think of it primarily as two separat...
Amazon Athena lets you run standard SQL against data sitting in S3, using the Glue Data Catalog for schema, with no clus...
Amazon Redshift is AWS's analytical data warehouse, available both as provisioned clusters (you manage cluster size, pay...
Kinesis Data Streams ingests continuous, real-time event data - think a live feed of PhonePe transaction events - that c...
General IAM concepts - what a policy is, what a role is - belong in the Cloud Engineering Mastery roadmap. What matters ...
For orientation only: Google Cloud Storage plays the same role as S3, BigQuery combines the roles of Athena and Redshift...
General Terraform syntax - providers, resources, state, modules - is covered in the Cloud Engineering Mastery roadmap's ...
Every piece covered in this module fits into one continuous system. Here is the full picture, end to end. Source System ...
A teammate reports that a query filtering for a single day of Flipkart orders is scanning the entire dataset and costing...
Create an S3 bucket with the three-zone structure (raw, processed, curated), and upload a sample day of order data as CS...
Service / Concept What It Is For S3 raw/processed/curated Zone structure - immutable source, cleaned data, business-read...
Storing data in S3 without any partitioning strategy forces Athena to scan the entire dataset on every single query, reg...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.