An engineer at acme-payments ships a Docker image for the payments-api service. It needs to run continuously behind a load balancer, hold a predictable amount of memory, and keep a long-lived connection pool to Postgres open. The same team also needs a function that runs for 200 milliseconds every time a file lands in an S3 bucket - spinning up a container and keeping it warm all day for that would waste money for nothing. Neither compute model is wrong. They solve different shapes of problem. A service that needs to run continuously, hold long-lived connections, or requires predictable resource allocation is often a strong fit for ECS/Fargate. A short-lived, event-driven workload that benefits from automatic scaling and scaling to zero is often a strong fit for Lambda. Picking the wrong one shows up months later as either a huge idle compute bill or a service straining against a 15-minute execution limit it was never designed to work within. This module builds both compute models end to end - containers on ECS Fargate, functions on Lambda, the API Gateway front door that exposes Lambda to the internet, and the decision framework for choosing between them on your next project.
If you already know Docker, ECS adds very little that is genuinely new - most of it maps directly onto commands you already run. * A **Task Definition** is your `docker run` command, saved as versioned AWS config - which image, how much CPU and memory, which ports, which IAM roles * A **Task** is one running container started from that Task Definition - the ECS equivalent of a single `docker run` execution * A **Service** keeps a target number of tasks running continuously, restarting any that crash - the ECS equivalent of `docker-compose` keeping containers alive, except it also integrates with a load balancer ```json { "family": "acme-payments-api", "requiresCompatibilities": ["FARGATE"], "networkMode": "awsvpc", "cpu": "512", "memory": "1024", "containerDefinitions": [ { "name": "payments-api", "image": "123456789012.dkr.ecr.ap-south-1.amazonaws.com/payments-api:latest", "portMappings": [{ "containerPort": 8080 }] } ] } ``` > **Note:** `cpu` and `memory` here are for the whole Task, not per > container. `512` CPU units equals 0.5 vCPU - Fargate bills you for > exactly this much, whether the container is busy or idle. ### Choosing Fargate vs the EC2 launch type * **Fargate** - you define the container's CPU and memory, AWS runs it on infrastructure you never see. No servers to patch, no cluster capacity to plan. Scaling means changing task count, nothing else. * **EC2 launch type** - you provision the EC2 instances that form the cluster yourself, and ECS places containers onto them. You own patching, instance sizing, and cluster capacity planning in exchange for access to specific instance types, GPUs, or Reserved Instance pricing. > 💡 **Tip:** Default to Fargate for new workloads. Reach for the EC2 > launch type only when you have a concrete reason - GPU instances, deep > Reserved Instance savings already committed, or a compliance requirement > around physical instance isolation.
This is the single most confused pair of concepts in ECS, and it comes up in nearly every production incident involving permissions. * The **Execution Role** is used by the ECS agent itself, before your container code ever runs - to pull the image from ECR and to push logs to CloudWatch. Your application code never touches this role. * The **Task Role** is used by your application code while it runs - to call S3, DynamoDB, or any other AWS API your container needs. ECS Task starting up | Execution Role -> pulls image from ECR Execution Role -> creates the CloudWatch log stream | Container starts running | Task Role -> your app code calls S3, DynamoDB, SQS, etc. > 📌 **Remember:** If a task fails to even start, with an error about > pulling the image or writing logs, the Execution Role is missing a > permission. If the task starts fine but your application code gets an > `AccessDenied` calling S3 or DynamoDB, the Task Role is missing a > permission. Checking the wrong role wastes the first ten minutes of most > ECS permission incidents. ```json { "family": "acme-payments-api", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::123456789012:role/paymentsApiTaskRole", "containerDefinitions": [ { "name": "payments-api" } ] } ``` > 🔴 **Common Mistake:** Engineers attach S3 or DynamoDB permissions to the > Execution Role instead of the Task Role, assuming "it's the role on the > task definition" is enough. The container's own API calls run under the > Task Role specifically - putting application permissions on the > Execution Role does nothing for your app code, and leaves the actual > Task Role under-permissioned.
An ECS Service on its own has no public entry point. Traffic reaches it through an Application Load Balancer, which also handles the part of container networking that used to require manual bookkeeping. ### Why dynamic port mapping matters Running several containers on one host used to mean tracking which container landed on which random port by hand. With the `awsvpc` network mode Fargate uses, every task gets its own elastic network interface with its own private IP, so this problem mostly disappears - but the same underlying mechanism still applies when tasks register with a Target Group. Each task registers itself and its assigned port automatically. When a task stops, the ALB removes it from rotation immediately. ALB | Target Group: payments-api-tg | +-- Task 1 (10.0.1.12:8080) -- healthy -- receiving traffic +-- Task 2 (10.0.1.45:8080) -- healthy -- receiving traffic +-- Task 3 (10.0.1.51:8080) -- draining -- being replaced Health checks are configured on the Target Group, not on the Service or the Task Definition. A task only starts receiving traffic once its health check passes, and the ALB stops routing to it the instant a check fails. ### End-to-end deployment - Docker build to running Fargate service > **Note:** Prerequisites for this deployment: a VPC with private subnets, > an ALB with a listener and target group, security groups allowing the > ALB to reach the tasks, and the Execution Role and Task Role from the > previous section must already exist. This section wires a container > image into that infrastructure - it does not create the infrastructure > itself. ```bash ## Authenticate Docker to ECR - token is valid for 12 hours aws ecr get-login-password --region ap-south-1 | \ docker login --username AWS --password-stdin \ 123456789012.dkr.ecr.ap-south-1.amazonaws.com ## Build the image from the Dockerfile in the current directory docker build -t payments-api:latest . ## Tag it with the full ECR repository URI docker tag payments-api:latest \ 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payments-api:latest ## Push - this is the image the Task Definition above references docker push 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payments-api:latest ``` ```bash ## Register the Task Definition from the JSON shown earlier aws ecs register-task-definition \ --cli-input-json file://payments-api-taskdef.json \ --region ap-south-1 ## Create the Service - keeps 2 tasks running, wired to the ALB target group aws ecs create-service \ --cluster acme-payments-cluster \ --service-name payments-api-service \ --task-definition acme-payments-api \ --desired-count 2 \ --launch-type FARGATE \ --load-balancers targetGroupArn=arn:aws:elasticloadbalancing:ap-south-1:123456789012:targetgroup/payments-api-tg/abc123,containerName=payments-api,containerPort=8080 \ --network-configuration 'awsvpcConfiguration={subnets=[subnet-0a1b2c],securityGroups=[sg-0d4e5f],assignPublicIp=DISABLED}' \ --region ap-south-1 ``` > **Note:** `assignPublicIp=DISABLED` keeps the tasks in a private subnet, > reachable only through the ALB - the same pattern you would use for any > EC2-based web tier sitting behind a load balancer.
Lambda is AWS's serverless compute service. You write a function, tell AWS what triggers it, and AWS runs it on demand - no servers to provision, no capacity to plan ahead of time. You pay for the milliseconds your code actually executes, not for idle time. EC2 / ECS - always on: Lambda - on demand: Runs continuously Event arrives | | You pay even when idle Lambda starts, runs, returns | | You manage capacity Shuts down - pay only for that time Every Lambda invocation is bound by two hard limits that decide whether Lambda is even the right tool for a given task: * **Maximum execution time is 900 seconds (15 minutes).** Anything longer belongs on ECS, Fargate, or EC2 - there is no way to extend this. * **Memory is the only performance knob, from 128 MB to 10 GB.** Lambda has no separate CPU setting - allocating more memory proportionally increases the CPU and network bandwidth available to your function. > 📌 **Remember:** If a task might run past 15 minutes, or needs a > persistent process holding state between requests, Lambda is the wrong > tool from the start. Reach for ECS or Fargate instead of trying to work > around the timeout.
Lambda code never runs on its own - something has to invoke it. The trigger you choose also decides how Lambda behaves when it fails. | Trigger | Typical Use | Invocation Type | |:---|:---|:---| | API Gateway | Public HTTP APIs | Synchronous - caller waits | | S3 | React to a file upload | Asynchronous - S3 moves on | | SQS | Process queued messages | Poll-based - Lambda pulls batches | | EventBridge | Scheduled jobs, event routing | Asynchronous | | DynamoDB Streams | React to a table change | Poll-based - ordered per shard | The distinction between synchronous and asynchronous matters most when something goes wrong: * **Synchronous** (API Gateway, an SDK call, an ALB) - the caller is actively waiting. If Lambda throttles or errors, the caller gets the error back immediately and must decide whether to retry. * **Asynchronous** (S3, SNS, EventBridge) - the caller fires the event and moves on. Lambda retries automatically for a period of time, and if every retry fails, the event lands in a **Dead Letter Queue** for you to inspect later. * **Poll-based** (SQS, DynamoDB Streams, Kinesis) - Lambda itself polls the source and pulls batches of records to process, rather than being pushed to directly. ```python import json def lambda_handler(event, context): """ Triggered by S3 whenever a new object is created in the upload bucket. event['Records'] can contain more than one file if several uploads happened close together - always loop, never assume a single record. """ for record in event["Records"]: bucket = record["s3"]["bucket"]["name"] key = record["s3"]["object"]["key"] print(f"Processing {key} from {bucket}") # Real logic - generate a thumbnail, extract metadata, etc. return {"statusCode": 200, "body": json.dumps({"processed": len(event["Records"])})} ``` > 💡 **Tip:** For SQS specifically, set the queue's visibility timeout to > at least 6 times your Lambda's timeout. If Lambda is still processing a > message when visibility expires, SQS makes the message visible again and > a second Lambda invocation can pick up the same message - a subtle bug > that shows up as duplicate processing under load.
An engineer at acme-payments ships a Docker image for the payments-api service. It needs to run continuously behind a lo...
If you already know Docker, ECS adds very little that is genuinely new - most of it maps directly onto commands you alre...
This is the single most confused pair of concepts in ECS, and it comes up in nearly every production incident involving ...
An ECS Service on its own has no public entry point. Traffic reaches it through an Application Load Balancer, which also...
Lambda is AWS's serverless compute service. You write a function, tell AWS what triggers it, and AWS runs it on demand -...
Lambda code never runs on its own - something has to invoke it. The trigger you choose also decides how Lambda behaves w...
Lambda shuts an instance down after it finishes handling requests. The next invocation on a fresh instance has to load y...
By default, Lambda runs inside an AWS-owned VPC that has no route into your own private network. Public AWS services lik...
A Lambda function has no public URL of its own. To call it from a browser or mobile app, you put API Gateway in front of...
Every container running on ECS in this module pulls its image from Amazon ECR - AWS's private container registry with IA...
None of these three replace each other - they fit different traffic and duration shapes. Signal Best Fit Long-running pr...
This lab builds both patterns from this module against the same application - an ECS Fargate service behind an ALB, and ...
Task Command Push an image to ECR docker push <ecr-uri>:<tag> Register a Task Definition aws ecs register-task-definitio...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.