A Razorpay payments team ships a new feature. Within a month, product wants a dashboard showing conversion funnel drop-off by city, device, and payment method, refreshed every hour. The data already exists, scattered across DynamoDB tables, application logs in S3, and a Postgres read replica. Nobody can answer the question today because the data was never built to be queried together. This is the job. Not writing Lambda functions, not tuning EC2 instances. Moving data from where it is produced to where it can be queried, cheaply, reliably, and in a shape that does not fall over the day the dataset triples in size. > 📌 **Remember:** almost every AWS data service is solving the same underlying problem > in a different way: how do you process more data than fits on one machine, without > paying for compute you are not using. Keep asking "what problem does this solve that S3 > alone cannot" as you move through this module. ### The one picture to hold in your head Before touching any service, look at the shape of the whole pipeline. Every section in this module is one box in this diagram. If you get lost later, come back here. Applications, databases, logs | v Ingestion (streaming or batch) | v Raw S3 data lake | v Catalog + transformation | v Processed / curated S3 data | v Athena or Redshift | v Dashboards Six ideas, six parts of this module. Ingestion gets data in. The data lake stores it cheaply. The catalog and transformation step makes it queryable and efficient. Athena and Redshift answer questions against it. QuickSight or Grafana turns answers into dashboards. Iceberg, covered last, is a quality upgrade to the data lake box, not a separate stage of the pipeline. > 💡 **Tip:** every time a new service is introduced in this module, ask yourself which > box in this diagram it belongs to. That habit will stop the individual sections from > feeling like a disconnected list of AWS trivia.
Real-time data arrives from somewhere: a mobile app sending click events, a payment service emitting transaction records, IoT sensors reporting temperature every second. Before any of it can be queried, it has to be captured. This is the first box in the diagram above. ### Kinesis Data Streams in plain English first **Simple explanation:** imagine a shared notebook. Multiple apps write entries into it in the order they happen. Multiple readers can each flip through the notebook at their own pace, and any reader can flip back to an earlier page and re-read something they already saw. **Technical definition:** Kinesis Data Streams is a durable, ordered, replayable buffer. "Durable" means the data is safely retained even after it is read. "Ordered" means records keep their sequence within a shard. "Replayable" means a consumer can re-read older records instead of only seeing new ones as they arrive. **Real-world example:** a Zerodha trade-execution service writes every trade as it happens into a Kinesis stream. A fraud-detection service and a real-time-analytics service both read from that same stream independently, each at their own speed, without either one affecting the other. **Decision rule:** use Kinesis Data Streams when more than one consumer needs to read the same events, or when you need the ability to replay history. ### Firehose in plain English first **Simple explanation:** think of Firehose as a pipe, not a notebook. Data goes in one end and comes out the other end already delivered to a destination. There is nothing to read from in the middle, because there is no "middle" to read from. **Technical definition:** Kinesis Data Firehose (renamed **Amazon Data Firehose** in 2024) batches incoming data, optionally transforms it, converts its format, and writes it to a destination such as S3, Redshift, or OpenSearch. You do not write any consumer code. **Decision rule:** use Firehose when your only goal is "get this data into S3 or Redshift," with no custom processing logic and no second application that needs to read the same raw events. > 💡 **Tip:** if you are only capturing events and dropping them into storage, you often > do not need Kinesis Data Streams at all. Firehose alone, writing directly from your > application, is simpler and cheaper. ### MSK in plain English first **Simple explanation:** MSK is AWS running Apache Kafka servers for you, so you get the real Kafka experience without racking your own servers. **Technical definition:** Amazon MSK (Managed Streaming for Apache Kafka) provides managed Kafka brokers. <cite index="23-1">Kafka's consumer group model lets multiple independent applications consume the same topic from different positions, each tracking their own offset, which is more flexible than Kinesis's checkpointing model for complex fan-out architectures with heterogeneous consumers.</cite> **Decision rule:** <cite index="20-1">if existing producers, consumers, and Kafka Connect connectors need to work with zero code changes, MSK is the clear answer, since Kinesis would require rewriting every service to use the AWS SDK.</cite> ### Putting the three side by side <cite index="20-1">As a rough cost rule of thumb, Kinesis is cheaper below about 5 MB/s of throughput, while MSK becomes more cost-effective above about 50 MB/s once you are running provisioned brokers on Graviton instances.</cite> But in an interview or in a real design review, throughput is rarely the deciding factor. Ask these three questions instead: * Do more than a handful of independent applications need to read the same stream at full speed? That points to MSK or Kinesis Data Streams, not Firehose. * Does the team already run Kafka, or are producers and consumers already written against the Kafka API? That points to MSK. * Is the goal simply "land this data in S3," with no replay and no second consumer ever expected? That points to Firehose. ```text Simplest AWS-native pattern (no Kafka expertise needed): App -> Kinesis Data Streams (on-demand mode) -> Firehose -> S3 (Parquet) Kafka-ecosystem pattern (existing Kafka investment): 20 microservices -> MSK topics -> consumer groups per team -> independent processing ``` > **Note:** "on-demand mode" for Kinesis Data Streams means AWS manages shard scaling > automatically, the same way DynamoDB on-demand removes the need to provision > read/write capacity yourself. Use on-demand mode unless you have a very predictable, > steady throughput pattern where provisioned shards work out cheaper. <cite index="26-1">In production, Data Streams, Firehose, and Managed Service for Apache Flink often work together as three separate jobs doing three separate things: Data Streams is the durable replayable stream, Firehose is managed delivery to a destination, and Flink is stateful processing on top.</cite> A common pattern: Kinesis Data Streams captures the raw event with replay capability, and Firehose is attached as one of its consumers purely to handle the S3 delivery, giving you both replayability and zero-code delivery in the same pipeline. > 🔴 **Common Mistake:** writing directly to Firehose because it is simpler, then > discovering six months later that a second team needs to consume the same events for > a different purpose. Firehose has no replay and no multi-consumer model. If there is > any chance a second consumer will exist later, put Kinesis Data Streams in front and > attach Firehose as a consumer of it, not the other way around.
This is the "Raw S3 data lake" box from the diagram at the top. Whether data arrived through Kinesis, Firehose, MSK, or a nightly batch export, it lands here first, as files in S3, before anything else happens to it. **Simple explanation:** a data lake is just a big, cheap folder in the cloud. Unlike a database, nothing forces the data to have a fixed shape when it lands. That flexibility is the whole point: you can dump data first and figure out its structure later. Most teams organize the lake into zones inside the same bucket, or across buckets, to keep raw and finished data clearly separated. * **Raw zone** - data exactly as it arrived, untouched, kept for auditing and replay * **Processed zone** - cleaned, converted to an efficient format like Parquet, still fairly close to the source structure * **Curated zone** - business-ready tables, joined and aggregated, what dashboards actually query > 📌 **Remember:** never let a dashboard or analyst query the raw zone directly. Raw > data is for reprocessing, not for daily querying. Every query-facing tool in this > module should point at the processed or curated zone.
Files sitting in S3 do not know their own schema. Something needs to look at the files and register "this folder holds a table with these columns and these types" so that SQL tools can find and query it. That is the job of the Glue Data Catalog and the Glue Crawler. **Simple explanation:** a crawler is like a librarian who walks through a warehouse of unlabeled boxes, opens each one, and writes a label describing what is inside and where it sits on the shelf. The librarian does not move any boxes. They only write labels. **Technical definition:** <cite index="11-1">a Glue crawler populates the Glue Data Catalog with databases and tables, and can crawl multiple data stores in a single run, creating or updating tables in the Data Catalog on completion.</cite> <cite index="11-1">When the crawler runs, it evaluates classifiers to infer the schema, using built-in classifiers for common formats like JSON, CSV, and Apache Avro, or custom classifiers you define yourself.</cite> > 📌 **Remember:** the crawler does not move or copy your data. It only reads enough of > it to infer a schema and registers that schema as a table definition. Your data stays > exactly where it is in S3. The **Glue Data Catalog** itself is the shared, central metadata store this crawler writes into. Athena, Redshift Spectrum, and EMR all read table definitions from this one catalog. Build it once, and every query engine on your data lake understands the same tables. S3 (raw) -> Glue Crawler -> Glue Data Catalog | v Glue ETL Job | v S3 (processed, Parquet)
Once data is catalogued, it usually still needs work: cleaning, joining, converting CSV to Parquet, aggregating. Three AWS tools can do this, and picking the wrong one is one of the most common expensive mistakes in AWS data engineering. **Simple explanation of the choice:** Glue ETL is a kitchen you rent by the minute - you only pay while you are cooking. EMR on EC2 is a kitchen you rent by the month - you pay whether you are cooking or not, but you get full control over every appliance in it. | Tool | Best for | Cost model | |:---|:---|:---| | AWS Glue ETL | Serverless jobs, unpredictable schedules, teams without cluster ops | Pay per DPU-hour while the job runs | | EMR on EC2 | Large, continuous, predictable workloads with deep Spark/Hadoop tuning needs | Pay for EC2 instances whether jobs are running or not | | EMR Serverless | Spark/Hadoop workloads without cluster management overhead | Pay per vCPU/memory-second while the job runs | <cite index="6-1">AWS EMR is ideal for heavy big data and ML workloads needing full Spark or Hadoop cluster control, and is often cheaper than Glue at scale, but requires more configuration and ongoing operations.</cite> <cite index="8-1">The core trade-off with EMR on EC2 is that once you turn the cluster on, it stays on and you pay for those EC2 machines whether the job is running or not, essentially renting the underlying Linux machines by the hour.</cite> <cite index="8-1">The teams who use EMR on EC2 successfully are usually the ones with deep big data expertise already, people who have run on-premises Hadoop or Spark clusters for years and know exactly what configuration they need.</cite> If that is not your team yet, start with Glue ETL or EMR Serverless, and only move to EMR on EC2 when a specific, measured cost or performance need justifies the extra operational load. > 🔴 **Common Mistake:** spinning up a full EMR on EC2 cluster for a nightly job that > processes 2GB of data and finishes in four minutes. The cluster still bills for > provisioning and idle time around that four minutes. Glue ETL or EMR Serverless would > cost a fraction as much for the same output, because you only pay while work is > actually happening. ```python # AWS Glue ETL job (PySpark) - converts raw CSV order events to partitioned Parquet # Runs as a serverless Glue job, no cluster to manage import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from awsglue.context import GlueContext from awsglue.job import Job from pyspark.context import SparkContext # Standard Glue job bootstrap - reads job name passed in at run time args = getResolvedOptions(sys.argv, ["JOB_NAME"]) sc = SparkContext() glueContext = GlueContext(sc) spark = glueContext.spark_session job = Job(glueContext) job.init(args["JOB_NAME"], args) # Read raw order events from the Glue Data Catalog table # This table was created automatically by the Glue Crawler raw_orders = glueContext.create_dynamic_frame.from_catalog( database="swiggy_raw", table_name="order_events_csv" ) # Convert to Spark DataFrame for easier transformation logic df = raw_orders.toDF() # Write out as partitioned Parquet - partitioning by order_date matches # how the analytics team will query this data (WHERE order_date = ...) df.write.mode("append") \ .partitionBy("order_date", "city") \ .parquet("s3://swiggy-datalake-prod/orders/processed/") job.commit() ``` > **Note:** `DynamicFrame` is a Glue-specific data structure, similar to a Spark > DataFrame but more tolerant of schema inconsistencies in raw data, like a column that > is sometimes a string and sometimes null. Convert to a regular DataFrame with > `.toDF()` once your data is clean enough for standard Spark operations.
This is the box in the diagram where a human, or a dashboard, finally asks a question of the data. ### Making Athena cheap instead of expensive **Simple explanation:** Athena lets you run SQL directly against files in S3, with no server to set up. You are billed only for the amount of data your query actually reads, similar to a taxi meter that charges by distance travelled rather than a flat monthly fare. **Technical definition:** <cite index="14-1">Athena costs 5 dollars per terabyte of data scanned.</cite> That single fact drives almost every optimization decision below, because the exact same query can cost 100 times more or less purely based on how the underlying data is stored. **Partitioning - the highest-leverage optimization.** <cite index="14-1">Registering partitions in the Glue Data Catalog lets Athena automatically apply partition pruning for queries with matching WHERE clause filters.</cite> <cite index="19-1">If your data is partitioned by date and you query for one day, Athena reads only that day's partition instead of the whole dataset - for a year of daily partitions, that is roughly one three-hundred-and-sixty-fifth of the data scanned.</cite> ```sql -- Without partition filter: scans the ENTIRE table, every day of history SELECT event_type, COUNT(*) FROM analytics.events_partitioned GROUP BY event_type; -- With partition filter: scans only 2026-05-01's data SELECT event_type, COUNT(*) FROM analytics.events_partitioned WHERE year = '2026' AND month = '05' AND day = '01' GROUP BY event_type; ``` > 📌 **Remember:** partition design should mirror your most common query filter, not > your data source structure. Date is almost always the first partition level, because > almost every dashboard query filters by a time range. Region, tenant, or status are > common second levels underneath date. **Columnar format and compression - the second highest-leverage optimization.** <cite index="12-1">Converting CSV data lakes to Parquet with Glue ETL jobs typically reduces Athena costs by 70 to 90 percent, and Snappy compression adds another 2 to 4 times reduction on top of the columnar layout.</cite> This works because Parquet is columnar: a query that only needs three columns out of thirty only reads those three columns off disk, while CSV forces a full row-by-row read regardless of how many columns you actually select. ```sql -- CTAS: the fastest way to migrate an existing CSV table to optimized Parquet CREATE TABLE analytics.events_parquet WITH ( format = 'PARQUET', parquet_compression = 'SNAPPY', external_location = 's3://swiggy-datalake-prod/events-parquet/', partitioned_by = ARRAY['event_date'] ) AS SELECT * FROM analytics.events_csv; ``` <cite index="14-1">This single CTAS statement transforms an entire raw dataset into a partitioned, Parquet-formatted, Snappy-compressed optimized table, and every future query against the optimized table runs at a fraction of the original cost.</cite> **Athena workgroups - isolating cost per team.** A workgroup separates query execution, billing, and limits between teams sharing the same account. Without one, a single accidental full-table scan can consume an entire team's monthly budget in one query. ```bash ## Create a workgroup with a hard 10GB per-query scan limit ## Any query exceeding this is automatically cancelled before it finishes aws athena create-work-group \ --name "analytics-team-queries" \ --configuration '{ "ResultConfiguration": { "OutputLocation": "s3://swiggy-athena-results/analytics-team/" }, "BytesScannedCutoffPerQuery": 10737418240, "EnforceWorkGroupConfiguration": true }' ``` <cite index="19-1">Any query that tries to scan more than the configured limit in that workgroup gets automatically cancelled.</cite> Set the equivalent property via CloudFormation for the same effect through infrastructure as code, and monitor the `AthenaDataScanned` CloudWatch metric with an alarm as a softer warning before hard limits kick in. > ⚠️ **Security:** workgroups are also a cost-allocation tool, not just a technical > limit. <cite index="14-1">Tag each workgroup and filter by tag in Cost Explorer for team-level cost attribution</cite> - without this, a single shared Athena bill makes it impossible to know which team's queries are actually driving the cost. ### Choosing between Athena and Redshift **Decision rule, stated simply:** if the question changes every time, use Athena. If the same question gets asked every single day against the same well-known dataset, use Redshift. * **Athena** is the right choice for ad-hoc, exploratory, infrequent queries against a data lake where you do not want to pre-provision anything. You pay per query, there is nothing to manage, and it scales to zero when nobody is querying. * **Redshift** is the right choice when the same complex queries run repeatedly against a relatively fixed, well-understood dataset. A dedicated, warmed-up cluster with optimized storage for repeated joins will consistently outperform and outcost Athena at that usage pattern. ```text Use Athena when: Use Redshift when: * Query patterns vary widely * Same complex queries run daily/hourly * Data lands in S3 from many sources * Dataset is curated and stable * Usage is bursty or unpredictable * Sub-second dashboard latency matters * Team wants zero infrastructure * Heavy joins across large fact tables ``` > 💡 **Tip:** many production teams run both. Athena queries the raw and lightly > processed layers of the data lake for exploration, while a curated subset of that same > data gets loaded into Redshift specifically to power the always-on BI dashboards that > the business checks every morning. ### QuickSight as the final box in the diagram QuickSight sits on top of both Athena and Redshift as AWS's native business intelligence tool, letting non-technical stakeholders build dashboards without writing SQL. Choose QuickSight when your primary audience is business users who need drag-and-drop chart building and AWS-native user management. Choose a tool like Grafana instead when your primary audience is engineers who already live there for infrastructure monitoring and want business metrics in the same place.
A Razorpay payments team ships a new feature. Within a month, product wants a dashboard showing conversion funnel drop-o...
Real-time data arrives from somewhere: a mobile app sending click events, a payment service emitting transaction records...
This is the "Raw S3 data lake" box from the diagram at the top. Whether data arrived through Kinesis, Firehose, MSK, or ...
Files sitting in S3 do not know their own schema. Something needs to look at the files and register "this folder holds a...
Once data is catalogued, it usually still needs work: cleaning, joining, converting CSV to Parquet, aggregating. Three A...
This is the box in the diagram where a human, or a dashboard, finally asks a question of the data. Making Athena cheap i...
This part is not a new stage in the pipeline. It is a quality upgrade to the "raw S3 data lake" and "processed data" box...
This lab rebuilds every box in the diagram from the top of the module, in order, and proves the cost impact of partition...
Decision Choose this Not this, when Real-time capture, AWS-native Kinesis Data Streams You need Kafka Connect or existin...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.