Data Engineering on AWS
Learn to design AWS data pipelines that do not bankrupt the company - Kinesis, Glue, Athena, Redshift, and Iceberg explained with real cost numbers.
What You'll Learn
Understanding Why Data Engineering Is Its Own Specialization
A Razorpay payments team ships a new feature.
Part 1 - Getting Data In: Ingestion
Real-time data arrives from somewhere: a mobile app sending click events, a payment service emitting transaction records, IoT sensors reporting...
Part 2 - The Data Lake: Where Everything Lands
This is the "Raw S3 data lake" box from the diagram at the top.
Part 3 - Making the Lake Queryable: Catalog and Crawlers
Files sitting in S3 do not know their own schema.
Part 4 - Turning Raw Data Into Something Cheap to Query: Transformation
Once data is catalogued, it usually still needs work: cleaning, joining, converting CSV to Parquet, aggregating.
Part 5 - Asking Questions: Athena, Redshift, and QuickSight
This is the box in the diagram where a human, or a dashboard, finally asks a question of the data.
Skills You'll Master
Curriculum Index9 topics
Understanding Why Data Engineering Is Its Own Specialization
A Razorpay payments team ships a new feature.
Part 1 - Getting Data In: Ingestion
Real-time data arrives from somewhere: a mobile app sending click events, a payment service emitting transaction...
Part 2 - The Data Lake: Where Everything Lands
This is the "Raw S3 data lake" box from the diagram at the top.
Part 3 - Making the Lake Queryable: Catalog and Crawlers
Files sitting in S3 do not know their own schema.
Part 4 - Turning Raw Data Into Something Cheap to Query: Transformation
Once data is catalogued, it usually still needs work: cleaning, joining, converting CSV to Parquet, aggregating.
Part 5 - Asking Questions: Athena, Redshift, and QuickSight
This is the box in the diagram where a human, or a dashboard, finally asks a question of the data.
Part 6 - Upgrading the Lake: Apache Iceberg
This part is not a new stage in the pipeline.
Building the Hands-On Data Pipeline Lab
This lab rebuilds every box in the diagram from the top of the module, in order, and proves the cost impact of...
Quick Reference and Common Mistakes
Seven mistakes show up repeatedly when engineers first build AWS data pipelines.
Career Impact
Roles that use the skills in this module.
Data Engineer
Platform Engineer
Cloud Engineer
DevOps Engineer
Next Modules
Related Guides
Practice on the Coding Sheet
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding Sheet