A payments company in Bengaluru captures ten million transaction events a day. A data science team wants to build a fraud model on top of that data. A separate compliance team needs an audit trail queryable by SQL going back three years. None of these needs are solved by a single AWS service - they are solved by connecting several services into one coherent pipeline, and choosing the wrong connection point anywhere in that chain either costs the company real money or breaks the pipeline the day traffic doubles. ### The Three Tracks This Module Builds Toward This module is where AWS Core Services, Container Engineering, and Cloud Architecture and Cost Engineering come together into genuinely advanced work. You already know how to build a VPC, size compute, and design for resilience and cost. Here, you apply that judgment across three specialized tracks - Data Engineering, Machine Learning Infrastructure, and Hybrid Cloud and Advanced Serverless - along with the modern data lake and streaming patterns that sit underneath all three. > 📌 **Remember:** This module assumes the AWS Core Services, Container Engineering, and Cloud Architecture and Cost Engineering modules are already familiar ground. The services here are not simpler versions of things you know - they are the specialized layer built on top of what you know. ### How the Sections Build on Each Other The module follows one continuous thread: real-time ingestion feeds the data lake, the data lake feeds SageMaker's training data, and the multi-account and hybrid networking sections cover the infrastructure all of that runs on top of. Read it in order the first time through, then use the Specialization Track checkpoint near the end to decide where to go deeper. ---
Real-time data arrives from somewhere - IoT sensors, clickstream events, application logs - and the first architectural decision is always the same: how fast does this data need to be usable, and does it need transformation on the way? Think of it like three different delivery services for the same package. One gets it there in seconds with full tracking and lets multiple people track it independently (Kinesis Data Streams). One is slower but batches many packages together cheaply for a single drop-off point (Amazon Data Firehose). One actually opens the package and repacks its contents before delivery (Managed Flink). ### Kinesis Data Streams - Real-Time, You Manage the Consumers **Kinesis Data Streams** ingests data continuously into ordered, durable shards. Records are available to consumers within roughly one second of being written, and multiple independent applications can read the same stream at the same time without affecting each other. Producers (IoT devices, app servers) | Kinesis Data Streams (data organized into shards) | Multiple independent consumers read the same data: -> Lambda function processing events in real time -> Managed Flink application doing complex aggregation -> A custom application on EC2 You are responsible for writing the consumer application, whether that is a Lambda function, an application on EC2, or Managed Flink. This gives you full control over exactly what happens to each record and how quickly, at the cost of writing and operating that consumer yourself. ### Amazon Data Firehose - Near Real-Time, Fully Managed Delivery **Amazon Data Firehose** (formerly known as Kinesis Data Firehose) is not a stream you read from - it is a fully managed delivery service that batches incoming records and loads them into a destination: S3, Redshift, OpenSearch, or a third-party HTTP endpoint. There is no consumer application to write. Producers -> Data Firehose -> (optional Lambda transform) -> S3 / Redshift / OpenSearch A Firehose delivery stream is configured around a primary delivery destination, rather than providing the independent-consumer model of Kinesis Data Streams. Firehose buffers records before delivering a batch, based on a configurable buffer size and interval you control - this makes it near real-time rather than the sub-second latency of Data Streams, but the exact delivery lag depends on how that buffer is tuned, not a fixed number. > 🔴 **Common Mistake:** Expecting Firehose to behave like Kinesis Data Streams with multiple independent readers. A Firehose delivery stream is built around a primary delivery destination, not a general-purpose multi-consumer stream. If multiple independent applications need to consume the same event stream, use Kinesis Data Streams instead. ### Managed Flink - When the Data Needs Real Processing, Not Just Delivery **Amazon Managed Service for Apache Flink** runs actual stream-processing applications - windowed aggregations, joins across streams, complex event pattern detection - written in Java, Scala, or SQL, with AWS handling the underlying infrastructure. It reads from Kinesis Data Streams or Amazon MSK, never from Data Firehose. Kinesis Data Streams ──┐ ├──> Managed Flink (aggregate, join, detect patterns) ──> output Amazon MSK (Kafka) ──┘ > 🔴 **Common Mistake:** Assuming Flink can read directly from Amazon Data Firehose. It cannot - Flink consumes from Kinesis Data Streams and Amazon MSK only. If your pipeline sends data to Firehose first, Flink has no way to read it from there; the data needs to reach Flink through Data Streams or MSK instead. ### The Decision in One Table | Need | Service | |:---|:---| | Sub-second latency, multiple independent consumers | Kinesis Data Streams | | Simple delivery to S3/Redshift/OpenSearch, no custom logic | Amazon Data Firehose | | Real-time aggregation, joins, or pattern detection | Managed Flink (on Data Streams or MSK) | | Already running Kafka, need the broader Kafka ecosystem | Amazon MSK | > 📌 **Remember:** Data Streams gives you control and multiple consumers but you write the consumer code. Data Firehose gives you zero code but is built around a single delivery destination and no custom processing beyond a simple Lambda transform. Managed Flink is what you reach for when the processing itself, not just the delivery, is the hard part. ---
A data lake is not one service - it is a pattern where several services each do one job, and the value comes from how cleanly they hand off to each other. Getting this pattern right is one of the highest-leverage architecture skills in a data engineering role. ### The End-to-End Flow Raw data lands in S3 (the ingestion bucket, often via Kinesis Firehose) | AWS Glue Crawler scans it and detects schema | Glue Data Catalog stores the schema as metadata (a table definition) | Glue ETL job (Spark under the hood) converts CSV/JSON to Parquet, partitions it by date or other useful keys, writes to a curated bucket | Amazon Athena queries the curated data directly with SQL, reading the schema from the Glue Data Catalog | Amazon Redshift (via Spectrum, or after a load) handles the heavier, more complex BI queries that need joins and indexes | Amazon QuickSight builds dashboards on top of Athena or Redshift Every arrow in that chain matters. Glue Crawlers automate schema discovery and catalog updates for supported data sources, which is convenient - but they are not mandatory. Mature platforms sometimes manage important table schemas explicitly through infrastructure as code or a controlled schema pipeline instead of relying on automatic discovery for everything. Skip the Parquet conversion, though, and Athena scans far more data than it needs to on every query. Skip Lake Formation (when you have it) and you are managing access control separately for Athena, Redshift Spectrum, and EMR instead of once, centrally. ### Organizing the Lake into Zones A data lake without internal structure quickly becomes a dumping ground nobody trusts. The common pattern - often called a medallion architecture - organizes S3 into zones by how processed the data is: S3 │ ├── Raw / Bronze │ Original data, immutable, exactly as it arrived │ ├── Curated / Silver │ Cleaned, deduplicated, standardized, converted to Parquet │ └── Analytics / Gold Business-ready, aggregated datasets ready for BI and dashboards Each zone typically lives in its own S3 prefix or bucket, with its own Glue Data Catalog database, and access permissions get progressively more open as data moves from Bronze to Gold - raw data often has the tightest restrictions since it may still contain unmasked sensitive fields. ### Why Glue Data Catalog Is the Layer Everything Else Reads The Data Catalog is easy to underestimate because it does not "do" anything visible - it just stores metadata. But Athena, Redshift Spectrum, EMR, and Glue ETL jobs all read the same catalog to understand what is sitting in S3, which means a single Crawler run keeps every one of those services in sync automatically. Data sources (S3, RDS, DynamoDB, JDBC) | Glue Crawler scans and detects schema | Glue Data Catalog (one shared metadata layer) | Read by: Athena, Redshift Spectrum, EMR, Glue ETL jobs > 💡 **Tip:** Enable Glue Job Bookmarks on any ETL job that runs repeatedly on the same growing dataset. Without bookmarks, every run reprocesses the entire dataset from the beginning; with bookmarks enabled, each run only processes data that has arrived since the last successful run. ### Where Lake Formation Fits on Top of This Pattern **Lake Formation** does not replace Glue - it is built on top of it, adding centralized row-level and column-level access control that Athena, Redshift Spectrum, and EMR all respect consistently, instead of each service enforcing its own separate permission model. Without Lake Formation: IAM policies configured separately for Athena access, Redshift Spectrum access, EMR access -> hard to guarantee consistent row/column restrictions With Lake Formation: permissions defined once -> "Team X sees only rows where region = ap-south-1" -> "Finance sees the salary column, nobody else does" -> enforced identically across Athena, Redshift Spectrum, and EMR > **Note:** This is a simplified mental model of the outcome, not the exact mechanism. In practice, Lake Formation implements row and column-level security through data filters and LF-Tags (attribute-based access control) rather than one flat rule per team - the effect is the same centralized enforcement, but the underlying configuration involves defining filters and tagging resources and principals accordingly. > 📌 **Remember:** Use Glue alone when you just need ETL and a metadata catalog. Reach for Lake Formation specifically when you need centralized, fine-grained access control enforced consistently across multiple query engines - that consistency is the entire value proposition. ---
### How Columnar Format and Partitioning Cut Cost Together Athena charges based on the volume of data scanned per query, not on infrastructure you provision, which makes the cost model fundamentally different from a database you keep running around the clock. Athena's pricing is a rate per TB of data scanned - check the current AWS pricing page for the exact figure, since it can change, but the mental model is simple: Cost ≈ Data scanned × current price per TB CSV/JSON: reads the entire file even for a one-column query Parquet/ORC: reads only the columns the query actually needs Example: a 100 GB dataset, querying two columns out of twenty CSV -> scans close to the full 100 GB Parquet -> scans a fraction of that, since it is columnar This is why the Glue-to-Parquet conversion step in the data lake pattern above is not an optional optimization. Columnar formats can dramatically reduce the data Athena scans, especially when a query selects only a subset of columns, and the exact improvement depends on column selection, compression, partitioning, file sizes, and the specific query - but for suitable workloads, combining Parquet with effective partitioning commonly reduces both cost and latency by an order of magnitude. Partitioning compounds this further. A query filtered on `WHERE year=2024 AND month=01` against data partitioned by year and month in S3 folder structure never even opens the files outside that folder - the cost reduction from partitioning and columnar format together is multiplicative, not additive. ### Avoiding the Raw-Format Habit > 🔴 **Common Mistake:** Running Athena directly against raw CSV files "to get started quickly" and never revisiting that decision once the dataset grows. What looks like a negligible cost at gigabyte scale becomes a real line item at terabyte scale, and by then the queries, dashboards, and habits built around the raw format are harder to migrate away from than if Parquet had been the default from day one. ---
Amazon SageMaker exists for the moment none of AWS's pre-built ML services - Rekognition, Comprehend, Personalize - fit your specific problem, and you need to train a model on your own proprietary data. A production ML workflow is a loop, not a one-time pipeline, and conflating its stages is a common source of confusion in interviews and real projects alike. Data | Feature Engineering | Training Job -> trains a model, produces artifacts saved to S3 | Evaluation -> checks accuracy/quality against a held-out test set | Model Registry -> versions and tracks models, records approval status | Approval -> a human or automated gate promotes the model | Deployment (Real-Time Endpoint / Batch Transform / Serverless Inference) | Monitoring -> Model Monitor watches live predictions | Drift Detection -> flags when live data or quality diverges from training baseline | Retraining -> triggers a new pass through this same loop ### Training Job A training job runs your training code (built-in algorithm, your own script, or a custom container) against training data in S3, on infrastructure SageMaker provisions and tears down automatically. You choose the instance type, and for large models, GPU instances plus Spot Instances (via **Managed Spot Training**) can meaningfully reduce training cost - though GPU Spot capacity is often the least available and most frequently reclaimed instance type, so a training job relying on it needs to checkpoint frequently; without frequent checkpoints, an interrupted job can lose more progress than the discount was worth. ### Model Registry and the Approval Gate Once trained, a model is not automatically production-ready. The **Model Registry** is where trained model versions live, get evaluated against a held-out test set, and get explicitly approved before deployment - giving you a controlled promotion path instead of deploying whatever the latest training run happened to produce. ### Choosing a Deployment Target Real-time endpoint -> Need a prediction back in milliseconds, continuously -> Example: fraud scoring on a live transaction Batch transform -> Have a large dataset to score all at once, no urgency -> Example: scoring last month's entire customer base overnight Serverless inference -> Unpredictable or intermittent traffic, cost-sensitive -> Example: an internal tool used a few times a day > 📌 **Remember:** A real-time endpoint costs money every hour it exists, whether or not it is being called. Batch Transform and Serverless Inference both avoid that idle cost, in different ways - Batch Transform by running once and stopping, Serverless Inference by scaling to zero between invocations. Match the deployment target to the actual traffic pattern, not to whichever one sounds most impressive. ### Monitoring, Drift Detection, and Closing the Loop Deploying a model is not the end of the workflow - it is the point where the workflow needs to start watching itself. **SageMaker Model Monitor** continuously samples live prediction requests and responses from a deployed endpoint and compares them against a baseline captured from the training data. Two different things can drift, and they call for different responses: * **Data drift** - the live input data's statistical distribution has shifted away from what the model was trained on (a fraud model trained on pre-pandemic spending patterns, running against today's spending patterns). * **Model quality drift** - the model's actual prediction accuracy is degrading, measurable once ground-truth labels for past predictions become available. A drift alert is the trigger that closes the loop - it is what should kick off a retraining job, feeding fresh data back through the same Training Job -> Evaluation -> Registry -> Approval -> Deployment sequence, rather than a human noticing performance has quietly degraded months later. ### MLOps - Treating Models Like Software **MLOps** applies CI/CD discipline to the full loop above: automated retraining pipelines, drift detection wired to actually trigger those pipelines, endpoint autoscaling based on real traffic, and the same versioning and approval gates you would expect from any production software deployment. Model drift is not a hypothetical concern - left unmonitored, it is the default outcome over time as real-world data moves away from whatever the model was originally trained on. ---
A company migrating to AWS rarely migrates everything on day one. Data centers stay online for years, sometimes indefinitely, and connecting them to AWS securely and reliably is its own architecture decision with real trade-offs between cost, setup time, and bandwidth. ### The Three Options Compared | Option | Setup time | Bandwidth | Best for | |:---|:---|:---|:---| | Site-to-Site VPN | Minutes to hours | Limited by internet connection, encrypted over the public internet | Quick setup, lower bandwidth needs, backup connectivity | | Direct Connect | Weeks (physical circuit provisioning) | Dedicated, up to 100 Gbps, consistent low latency | High, sustained bandwidth needs, predictable latency requirements | | Transit Gateway | Minutes, but requires other connectivity already in place | Depends on what it is connecting | Hub-and-spoke for many VPCs and/or multiple VPN or Direct Connect connections | **Site-to-Site VPN** encrypts traffic over the public internet using IPsec tunnels - fast to set up, but bandwidth and latency are only as good as the underlying internet path, which makes it unsuitable as the sole connection for latency-sensitive, high-throughput workloads. **Direct Connect** is a dedicated, private physical network connection from your data center to an AWS Direct Connect location - not routed over the public internet at all. This gets you consistent bandwidth and latency, but the physical circuit takes real time to provision, so it is not the answer when you need connectivity today. **Transit Gateway** solves a different problem entirely - it is a hub that many VPCs, VPN connections, and Direct Connect connections can attach to, instead of a tangled mesh of individual VPC peering connections. It does not replace VPN or Direct Connect; it is the hub they plug into once you have more than a couple of VPCs or connections to manage. Without Transit Gateway (peering mesh): VPC-A <-> VPC-B <-> VPC-C <-> VPC-D -> N VPCs need roughly N*(N-1)/2 peering connections With Transit Gateway (hub and spoke): VPC-A, VPC-B, VPC-C, VPC-D -> all attach to one Transit Gateway -> N VPCs need N attachments, not N*(N-1)/2 connections > 💡 **Tip:** Transit Gateway is not free infrastructure - it charges per attachment-hour and per GB of data processed through it. For very high-throughput traffic between just two or three VPCs, a direct VPC peering connection can sometimes be cheaper than routing that same traffic through a Transit Gateway. The simplification Transit Gateway offers is architectural, not automatically financial - check both before assuming it is the cheaper option at your specific scale. > 🔴 **Common Mistake:** Treating Direct Connect and VPN as mutually exclusive alternatives instead of complementary. A common, resilient pattern is Direct Connect as the primary path for its bandwidth and latency characteristics, with a Site-to-Site VPN configured as an automatic backup path if the Direct Connect circuit fails - giving you the performance of one and the redundancy of the other. > 💡 **Tip:** Direct Connect and VPN connections both commonly use BGP (Border Gateway Protocol) to exchange routes dynamically between your network and AWS, rather than requiring static routes to be manually maintained on both sides as your network changes. ### Connecting Direct Connect to Many VPCs with Direct Connect Gateway A single Direct Connect connection terminates at one AWS region by default. **Direct Connect Gateway** is what lets that one physical connection reach multiple VPCs, potentially across multiple regions, instead of provisioning a separate physical circuit for every VPC you need to reach. On-Premises Data Center | Direct Connect (the physical, dedicated circuit) | Direct Connect Gateway (fans out to multiple destinations) | Transit Gateway (routing hub) | VPC-A VPC-B VPC-C Transit Gateway and Direct Connect solve different layers of the same problem and are commonly combined, not substitutes for each other: **Direct Connect is the dedicated physical connectivity path** from on-premises into AWS, while **Transit Gateway is the routing hub** that fans a single connection - whether VPN, Direct Connect, or both - out to many VPCs cleanly. Direct Connect Gateway is the piece that lets a Direct Connect connection attach to a Transit Gateway (or to multiple VPCs) in the first place. ---
A payments company in Bengaluru captures ten million transaction events a day. A data science team wants to build a frau...
Real-time data arrives from somewhere - IoT sensors, clickstream events, application logs - and the first architectural ...
A data lake is not one service - it is a pattern where several services each do one job, and the value comes from how cl...
How Columnar Format and Partitioning Cut Cost Together Athena charges based on the volume of data scanned per query, not...
Amazon SageMaker exists for the moment none of AWS's pre-built ML services - Rekognition, Comprehend, Personalize - fit ...
A company migrating to AWS rarely migrates everything on day one. Data centers stay online for years, sometimes indefini...
From Deciding on Multi-Account to Operationalizing It Cloud Architecture and Cost Engineering covered why multi-account ...
Cloud Architecture and Cost Engineering introduced Step Functions for the Saga pattern. At the scale of a real data plat...
Choosing a Specialization Track for Your Career Direction Everything in this module works together, but most engineers e...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.