A senior engineer at a Bengaluru fintech gets a Slack message on a Friday evening. The fraud detection model just started flagging 40% of transactions as suspicious. Nothing in the application code changed. No deploy happened. The model is the same file it was yesterday. This is not a bug in the traditional sense. There is no stack trace. The code runs fine. The problem is that the world changed and the model did not know. Customer spending patterns shifted after a festival sale, and the model - trained on data from three months ago - now sees normal transactions as anomalies. This is the core reason ML infrastructure is its own specialisation and not just "deploying a Python script." A regular application either works or throws an error. A machine learning model can be running perfectly, serving requests with zero errors, and still be silently wrong. You need infrastructure that watches for that, not just infrastructure that keeps the server up. This module teaches the AWS services and patterns that make ML systems reliable in production: when to use a pre-built API instead of training your own model, how SageMaker actually works end to end, which deployment mode fits which traffic pattern, how to catch a model going stale before customers notice, and how Bedrock changes the calculus for teams that do not want to train anything at all. > 📌 **Remember:** ML infrastructure engineering is not about building models. It is > about building the systems around models - the pipelines that feed them, the > endpoints that serve them, and the monitors that watch them. ### Why this is not the same job as data science A data scientist may focus primarily on developing and evaluating a model - choosing features, tuning hyperparameters, hitting an accuracy target in a notebook. ML infrastructure and cloud engineering focus on making that model reproducible, deployable, scalable, observable, secure, and cost-effective once it leaves the notebook. Getting a model from `model.pkl` on a laptop to an endpoint serving production traffic reliably, with monitoring and cost controls, is a distinct skill set - closer to distributed systems engineering than statistics. In many organisations these responsibilities overlap and are shared across both roles; the distinction here is about the skill set this module teaches, not a rigid boundary between job titles. ### Where this fits in the bigger AWS picture This builds directly on the compute, storage, and IAM concepts from earlier in the roadmap. A SageMaker training job is, underneath, an EC2 instance with a role attached running a container pulled from ECR, reading data from S3, exactly like the containers module covered. The specialisation here is the ML-specific layer on top - the parts that do not exist for a normal web application. ### The one picture to hold in your head Every decision in this module traces back to one question: what kind of problem is this, and what happens after a model exists? +----------------------+ | What problem? | +----------+-----------+ | +----------+-----------+-----------------+ | | | v v v Generic AI task Needs company Needs custom (image/text/ knowledge in structured speech) language prediction | | | v v v Pre-built AWS Bedrock + SageMaker API Knowledge Bases training (Rekognition, (RAG) | Textract, v Comprehend, Choose Transcribe) deployment mode | v Monitor for drift | v Evaluate and retrain when justified Keep this picture in mind through the rest of the module. Every section below fills in one branch of it. ---
The single most expensive mistake in ML infrastructure is building a custom model for a problem AWS already solved. Before anyone writes a line of training code, this decision has to be made correctly. ### The four pre-built AI services and what each one actually does AWS ships four services that solve extremely common ML problems with a single API call, no training data, and no model management: * **Rekognition** - image and video analysis: object detection, facial analysis, content moderation, celebrity recognition, text-in-image detection * **Textract** - extracting structured text and data from documents, including tables and forms, not just raw OCR * **Comprehend** - natural language processing: sentiment analysis, entity recognition, key phrase extraction, language detection, PII detection * **Transcribe** - speech to text, including speaker separation and custom vocabulary for domain-specific terms > 💡 **Tip:** If your problem sounds like "extract X from an image/document/audio/ > text," check whether a pre-built service already does it before scoping a > SageMaker project. A one-line API call that ships this week beats a three-month > training project that might not even beat the pre-built accuracy. ### The decision table | Your problem | Use this | Not this | |:---|:---|:---| | Detect unsafe or inappropriate visual content in uploads | Rekognition content moderation | Custom SageMaker model | | Detect objects, labels, faces, or text in an image | Rekognition | Custom SageMaker model | | Extract line items from a scanned invoice | Textract | Custom OCR + custom NER | | Classify support tickets as angry/neutral/happy | Comprehend sentiment | Custom SageMaker model | | Convert customer support calls to searchable text | Transcribe | Custom speech-to-text | | Detect fraud specific to your transaction patterns | SageMaker custom model | Rekognition (does not exist for this) | | Recommend products based on your catalogue and users | SageMaker custom model | Comprehend (wrong service entirely) | The pattern is simple: if the task is generic enough that thousands of companies need the exact same capability, AWS has almost certainly built it and trained it on far more data than any single company could gather. If the task depends on **your** proprietary data - your specific fraud patterns, your specific customers, your specific catalogue - no pre-built service can know that, and custom training is the only option. > 🔴 **Common Mistake:** Reaching for SageMaker because it feels like the "real" > engineering solution, when a Rekognition or Comprehend API call would solve the > problem in an afternoon with better accuracy than a rushed custom model. Custom > training is expensive in engineering time, compute cost, and ongoing maintenance. > Only pay that cost when the problem genuinely requires it. ### When pre-built services fall short Pre-built services are trained on general-purpose data. They will not know that in your specific product catalogue, "cold-pressed juice" and "detox drink" are the same category, or that your fraud patterns look nothing like a generic fraud dataset because your customer base skews toward a specific demographic and city. The moment accuracy on your specific data matters more than general capability, it is time to move to custom training. ---
SageMaker is Amazon's fully managed platform for building, training, and deploying custom machine learning models. The value proposition is that you do not manage GPU clusters, you do not write infrastructure code to distribute training across machines, and you do not build your own model-serving framework. You bring data and either a built-in algorithm or your own training code, and SageMaker handles the infrastructure. +------------------+ +------------------+ +------------------+ | S3 (Raw Data) | --> | SageMaker | --> | S3 (Trained | | training data | | Training Job | | Model Artifact) | +------------------+ +------------------+ +------------------+ | v +------------------+ | Managed GPU/CPU | | instances, spun | | up then torn down | +------------------+ This diagram shows the core loop: data comes from S3, SageMaker provisions compute temporarily to run the training job, and the resulting model artifact - the trained weights - lands back in S3. The compute only exists for the duration of training. > **Note:** A "training job" in SageMaker is not a persistent server. It is a > short-lived, managed compute job - similar in spirit to a Lambda function, except > it can run for hours or days and has access to GPU instances. AWS bills you only > for the time the job actually runs. ### Three ways to bring an algorithm to SageMaker SageMaker training jobs are not a single pattern - there are three distinct ways to supply the algorithm, and knowing which one you are looking at matters because the code you write looks different for each: * **Built-in algorithm containers** - AWS provides a ready-made container that already contains the algorithm implementation (XGBoost, Linear Learner, K-Means, and others). You supply data in the expected format and hyperparameters; there is no training script to write at all, because the training logic already lives inside AWS's container. * **Framework containers with your own entry-point script** - AWS provides a managed container for a framework like XGBoost, TensorFlow, PyTorch, or scikit-learn, but you supply your own `entry_point` script (for example `train.py`) that defines how training actually runs inside that managed environment. This is the middle ground: AWS manages the environment, you control the training logic. * **Bring Your Own Container (BYOC)** - you build and own the entire Docker image, including the base OS, all dependencies, and the training code, then push it to ECR yourself. SageMaker only provisions the compute and mounts your data; every other decision is yours. > 📌 **Remember:** Start as far left as possible - the pure built-in algorithm > container - unless you have a specific reason to move right. XGBoost's built-in > container alone solves a huge share of real business ML problems: fraud detection, > churn prediction, demand forecasting. Move to a framework container with a custom > `entry_point` when you need custom preprocessing or a specific training loop. > Reach for BYOC only when the framework containers cannot express what you need at > all, such as an unusual runtime dependency or a completely custom architecture. > **Note:** The `xgb_estimator` example below uses `entry_point="train.py"` - this > is the framework-container pattern, not the pure built-in algorithm container. > It is shown here because it is the most common real-world starting point: it gives > you AWS's managed XGBoost environment while still letting you control exactly how > the training script reads data and reports metrics. ### The full training job anatomy A SageMaker training job needs four things specified, and understanding each one demystifies what is otherwise a confusing API call: 1. **The algorithm source** - either an AWS-provided container URI for a built-in algorithm, or your own ECR image URI for a custom container 2. **Input data channels** - S3 paths for training data, and optionally separate paths for validation data, pointed to by named "channels" like `train` and `validation` 3. **Instance configuration** - instance type (`ml.m5.xlarge` for CPU workloads, `ml.p3.2xlarge` for GPU-heavy deep learning), instance count for distributed training, and the IAM role the job assumes 4. **Output location** - the S3 path where the trained model artifact (`model.tar.gz`) gets written when training completes ```python import sagemaker from sagemaker.xgboost import XGBoost ## The SageMaker session handles auth and default bucket/role resolution session = sagemaker.Session() ## XGBoost estimator - this is the "recipe" for the training job, ## it does not start anything until .fit() is called xgb_estimator = XGBoost( entry_point="train.py", ## your training script framework_version="1.7-1", ## XGBoost version to use instance_type="ml.m5.xlarge", ## CPU instance, fine for tabular data instance_count=1, ## single instance, no distributed training needed role="arn:aws:iam::123456789012:role/SageMakerExecutionRole-prod-mumbai", output_path="s3://razorpay-ml-models/fraud-detection/output", hyperparameters={ "max_depth": 5, ## tree depth, controls model complexity "eta": 0.2, ## learning rate "objective": "binary:logistic", ## binary classification: fraud or not "num_round": 100 ## number of boosting rounds } ) ## This actually launches the training job - SageMaker provisions the ## instance, downloads data, runs training, and tears down the instance xgb_estimator.fit({ "train": "s3://razorpay-ml-models/fraud-detection/train/", "validation": "s3://razorpay-ml-models/fraud-detection/validation/" }) ``` > **Note:** `.fit()` is a blocking call in this example - your script waits until > training completes. In production pipelines, training jobs are usually launched > asynchronously and their completion is tracked through EventBridge events or > Step Functions, not by blocking a script. ### Cutting training cost with Spot Instances Training jobs, especially deep learning ones, can run for hours and burn through GPU instance costs quickly. SageMaker supports Spot Instances for training, which can provide significant savings compared with On-Demand pricing - commonly cited at up to around 90%, though the actual figure depends on instance type, region, and current Spot capacity conditions. The catch: Spot instances can be interrupted with a two-minute warning when AWS needs the capacity back. For a training job that has been running for six hours, losing all progress to an interruption is unacceptable. The fix is checkpointing. > 🔴 **Common Mistake:** Treating Spot training without checkpointing as a pure cost > optimisation with no downside. It can silently waste hours of compute spend if > interruptions happen repeatedly near the end of long jobs, with nothing to show > for it - the savings only materialise if training can resume from where it left > off. ```python xgb_estimator = XGBoost( entry_point="train.py", instance_type="ml.m5.xlarge", instance_count=1, role="arn:aws:iam::123456789012:role/SageMakerExecutionRole-prod-mumbai", output_path="s3://razorpay-ml-models/fraud-detection/output", use_spot_instances=True, ## enables Spot pricing, often much cheaper max_run=3600, ## max training time in seconds max_wait=7200, ## max time including Spot interruption waits checkpoint_s3_uri="s3://razorpay-ml-models/checkpoints/fraud-detection/" ## checkpoints let training resume from the last saved point after ## an interruption instead of restarting from zero ) ``` > 🔴 **Common Mistake:** Enabling Spot training for cost savings without setting > `checkpoint_s3_uri`. An interruption at hour five of a six-hour job means the > entire five hours of progress is lost and training restarts from scratch - the > "savings" evaporate the moment one interruption happens on a long job. ---
Training produces a model artifact. Getting predictions out of that model in production - called **inference** - is a completely separate decision, and picking the wrong deployment mode is one of the most common and expensive mistakes in ML infrastructure. SageMaker offers three distinct ways to serve predictions, each suited to a different traffic pattern. > 💡 **Tip:** The deployment decision is really a question about your traffic > pattern: is it constant, occasional, or bursty? Answer that first, then pick the > mode. Do not default to real-time endpoints just because they feel like the > "standard" way to serve a model. ### Real-time endpoints A real-time endpoint is a persistent, always-on server behind an HTTPS API, returning predictions synchronously. This is what you use when an application is waiting on the response to continue - a fraud check that must complete before a payment is approved, a recommendation that must render before a page loads. * Designed for low-latency synchronous inference - actual latency depends on model size, instance type, payload size, and preprocessing, so measure it for your own model rather than assuming a fixed number * Billed continuously per hour the endpoint is running, whether or not it receives traffic * Best for steady, predictable traffic where the always-on cost is justified by constant use ```python ## Deploying a trained estimator directly to a real-time endpoint predictor = xgb_estimator.deploy( initial_instance_count=2, ## 2 instances for basic HA instance_type="ml.m5.large", endpoint_name="fraud-detection-prod-mumbai" ) ## Invoking the endpoint - this is what your application code calls response = predictor.predict(transaction_features) ``` ### Batch transform Batch transform processes a large dataset all at once, offline, with no persistent endpoint. You point it at an S3 input path, it runs predictions across every record, and writes results to an S3 output path, then the compute shuts down. * No ongoing cost - you pay only for the duration of the batch job * Best for high-throughput, non-urgent workloads: scoring your entire customer base overnight for churn risk, running fraud re-scoring across a week of historical transactions * Latency does not matter here because nothing is waiting on an individual response ```python transformer = xgb_estimator.transformer( instance_count=4, ## parallelise across 4 instances instance_type="ml.m5.xlarge" ) ## Scores every record in the input path, no endpoint stays running afterward transformer.transform( data="s3://razorpay-ml-models/fraud-detection/nightly-batch/", content_type="text/csv" ) ``` ### Serverless inference Serverless inference automatically provisions compute only when a request arrives, and scales to zero when idle. It sits between real-time and batch: you get an API-style endpoint like real-time inference, but pay nothing during idle periods. * No cost while idle - the defining advantage over real-time endpoints * Cold start latency - the first request after idle time takes longer while compute spins up * Best for intermittent, unpredictable traffic: an internal admin tool used a few times a day, a feature still in early rollout with low and sporadic volume > **Note:** This module focuses on these three common deployment patterns because > they cover the large majority of real-world traffic shapes. SageMaker also > supports additional inference options, including asynchronous inference for large > payloads that do not need an immediate response. The exact options available and > best suited to a given workload should be evaluated based on model size, scaling > behaviour, and traffic requirements at the time you deploy. ### The decision framework | Traffic pattern | Latency need | Use this mode | |:---|:---|:---| | Constant, high volume | Synchronous response required | Real-time endpoint | | Large dataset, no urgency | Minutes to hours acceptable | Batch transform | | Sporadic, unpredictable | Synchronous when active, idle otherwise | Serverless inference | | Constant but low volume | Synchronous response required | Real-time, scaled to 1 instance minimum | > 📌 **Remember:** The three modes are not about model accuracy or capability - the > exact same trained model can be deployed through any of the three. This decision > is purely about your traffic pattern and cost tolerance, made after training is > already complete. > 🔴 **Common Mistake:** Deploying every model to a real-time endpoint by default, > including ones invoked twice a day by an internal script. That endpoint bills for > 24 hours of uptime to serve two requests - serverless inference or even a > scheduled batch job would cost a fraction as much for identical results. ---
Return to the 2 AM scenario from the opening. The fraud model did not break in the traditional sense - it kept running, kept returning predictions, kept returning HTTP 200. But the world underneath it shifted, and nobody was watching for that shift. This is the operational reality that makes ML infrastructure different from normal application infrastructure: a system can be functioning perfectly and be silently wrong at the same time. **SageMaker Model Monitor** exists specifically to catch this, but it is not automatic just because an endpoint exists. Monitoring requires you to configure what data gets captured, establish a baseline, schedule the comparison job, and define how alerts are handled. Once that setup is in place, it continuously compares the data flowing into your live endpoint against the baseline and alerts when they diverge. ### The two kinds of drift * **Data drift** - the statistical properties of incoming requests change. Average transaction amounts rise after a festival sale, or a new customer segment starts using the product with different behaviour than the training data reflected. The model receives inputs unlike anything it learned from. * **Model quality drift** - the model's actual prediction accuracy degrades over time, measurable once ground-truth labels become available (for example, once you know which flagged transactions were genuinely fraudulent). +----------------+ +----------------+ | Training Data | | Live Endpoint | | (baseline) | | (captured) | +-------+--------+ +-------+--------+ | | v v +-----------------------------------+ | Model Monitor compares the two | | statistical distributions | +-----------------+-----------------+ | v +-------------------+ | CloudWatch alarm | | if drift exceeds | | threshold | +-------------------+ This diagram shows the monitoring loop: Model Monitor captures a sample of live traffic hitting the endpoint, compares its statistical shape against the training data baseline, and fires a CloudWatch alarm the moment the two diverge past an acceptable threshold - after you have configured the capture, baseline, and schedule that make this comparison possible. ```python from sagemaker.model_monitor import DefaultModelMonitor from sagemaker.model_monitor.dataset_format import DatasetFormat ## Step 1: capture a baseline from the training data - this defines ## "normal" that live traffic will be compared against my_monitor = DefaultModelMonitor( role="arn:aws:iam::123456789012:role/SageMakerExecutionRole-prod-mumbai", instance_count=1, instance_type="ml.m5.xlarge" ) my_monitor.suggest_baseline( baseline_dataset="s3://razorpay-ml-models/fraud-detection/train/train.csv", dataset_format=DatasetFormat.csv(header=True), output_s3_uri="s3://razorpay-ml-models/fraud-detection/baseline/" ) ## Step 2: schedule ongoing monitoring against the live endpoint, ## comparing captured traffic to the baseline every hour my_monitor.create_monitoring_schedule( monitor_schedule_name="fraud-detection-drift-check", endpoint_input="fraud-detection-prod-mumbai", output_s3_uri="s3://razorpay-ml-models/fraud-detection/monitor-reports/", statistics=my_monitor.baseline_statistics(), constraints=my_monitor.suggested_constraints(), schedule_cron_expression="cron(0 * ? * * *)" ## runs every hour ) ``` > **Note:** `suggest_baseline` runs a processing job that calculates the statistical > profile of every feature in your training data - mean, standard deviation, data > type, missing value rate. This profile becomes the yardstick every future > comparison uses. Without a baseline, Model Monitor has nothing to compare against. > 📌 **Remember:** Model Monitor does not fix drift automatically. It detects and > alerts. The response - retraining, rolling back, or investigating the underlying > cause - is still a human or pipeline decision made after the alert fires. ---
Model Monitor catches drift once a model is in production. But there is a second, quieter production failure that happens even before drift sets in: **training-serving skew**. This is when the model was trained using one definition of a feature, but the production system calculates that same feature differently. The same value - say, "customer's average transaction value over the last 30 days" - gets computed separately by the training pipeline and the real-time serving code. If those two implementations drift apart even slightly, the model behaves differently in production than it did during training, and nobody notices until accuracy quietly drops. **Feature Store** solves this by centralising feature computation and storage in one place, accessed through two distinct paths: * **Online store** - low-latency key-value access (single-digit milliseconds), used by real-time endpoints that need the latest feature values instantly during inference * **Offline store** - a full historical record in S3, queried through Athena, used when generating training datasets that need point-in-time correctness > 💡 **Tip:** Think of the online store as "what is true right now for this > customer" and the offline store as "what was true at every point in history for > every customer." Training needs the second. Real-time inference needs the first. > Feature Store keeps both in sync from a single feature definition. ---
A senior engineer at a Bengaluru fintech gets a Slack message on a Friday evening. The fraud detection model just starte...
The single most expensive mistake in ML infrastructure is building a custom model for a problem AWS already solved. Befo...
SageMaker is Amazon's fully managed platform for building, training, and deploying custom machine learning models. The v...
Training produces a model artifact. Getting predictions out of that model in production - called inference - is a comple...
Return to the 2 AM scenario from the opening. The fraud model did not break in the traditional sense - it kept running, ...
Model Monitor catches drift once a model is in production. But there is a second, quieter production failure that happen...
Some production models gradually lose relevance as data and real-world behaviour change, per the drift discussion above ...
Everything covered so far assumes you are training a model from scratch on your own data. Since 2023, a large share of n...
Every ML infrastructure decision in this module collapses into one flowchart a senior engineer runs through before writi...
Before you start Estimated cost: A single ml.m5.xlarge training job running for under 15 minutes plus an ml.m5.large rea...
Decision Choose this When Pre-built vs custom Rekognition/Textract/Comprehend/Transcribe Generic task, no proprietary da...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.