Is API Gateway, Step Functions, and Cognito - Serverless Application Layer free to learn on DevOps Network?

Yes - this topic, like everything on DevOps Network, is 100% free with no paywall or sign-up gate.

API Gateway, Step Functions, and Cognito - Serverless Application Layer

Q: What does the API Gateway, Step Functions, and Cognito - Serverless Application Layer topic cover?

Build secure serverless APIs with API Gateway, orchestrate multi-step workflows with Step Functions, and manage user authentication with Cognito User Pools.

What you will learn

What API Gateway is and the three endpoint types
How API Gateway connects to Lambda, HTTP backends, and AWS services directly
Stages, deployments, and how to manage dev vs prod from one API
API Gateway security — IAM, Lambda Authorizers, and Cognito User Pools
WebSocket APIs for real-time two-way communication
AWS Step Functions — orchestrating multi-step workflows without writing glue code
Standard vs Express workflows and when to use each
Amazon Cognito — User Pools vs Identity Pools and exactly what each does
How Cognito integrates with API Gateway for authentication
Row-level DynamoDB security using Cognito Identity Pools

Why this matters

At Razorpay, every payment API is fronted by API Gateway — it handles authentication, rate limiting, and routing before a single line of business code runs. At Swiggy, the order confirmation flow touches six services — fraud check, inventory reservation, kitchen notification, delivery assignment, email, and analytics. Orchestrating that with Step Functions means one visual workflow instead of six services each calling the next. At CRED, user login and session management runs entirely on Cognito — no custom auth server to build or maintain. These three services together form the backbone of most serverless application architectures on AWS.

What is Amazon API Gateway

API Gateway is a fully managed service that sits in front of your backend and handles everything an API needs — routing, authentication, rate limiting, caching, CORS, SSL termination, and monitoring — before traffic ever reaches your Lambda function or EC2 instance.

◈ DIAGRAM

Client (mobile app, browser, partner system)
    ↓ HTTP/HTTPS request
API Gateway
    → authenticates the request
    → checks rate limits
    → routes to the correct backend
    → returns the response
    ↓
Backend (Lambda, HTTP endpoint, AWS service directly)

Three endpoint types:

TEXT

Edge-Optimized (default):
  Requests routed through CloudFront edge locations
  Best for: global clients accessing from many locations
  SSL certificate must be in us-east-1
 
Regional:
  Deployed in one specific AWS region
  Best for: clients in the same region, or when you add CloudFront yourself
  SSL certificate in the same region as the API
 
Private:
  Only accessible from within your VPC via a VPC Endpoint
  Best for: internal microservice APIs, not exposed to internet at all

Remember
Edge-Optimized APIs use CloudFront under the hood. This means the SSL certificate must always be in us-east-1 regardless of where your API is deployed. This trips up engineers who put the certificate in ap-south-1 and wonder why HTTPS fails.

API Gateway Integrations — What It Can Connect To

Lambda Function:

The most common integration. API Gateway receives the HTTP request, converts it to a JSON event, and invokes the Lambda function. Lambda returns a JSON response. API Gateway converts it back to HTTP.

◈ DIAGRAM

GET /orders/ORD-001
    ↓
API Gateway converts to Lambda event:
{ "pathParameters": { "orderId": "ORD-001" }, "httpMethod": "GET" }
    ↓
Lambda function runs → queries DynamoDB → returns result
    ↓
API Gateway returns HTTP 200 with JSON body

HTTP Backend:

Proxy requests to any public HTTP endpoint — your on-premises server, another cloud, a partner API. API Gateway adds auth, rate limiting, and SSL in front.

AWS Service Direct Integration:

API Gateway can call AWS services directly without Lambda in the middle. Send a message to SQS, put an item in DynamoDB, start a Step Functions execution — all from API Gateway alone.

◈ DIAGRAM

POST /orders → API Gateway → SQS SendMessage (no Lambda needed)

Useful for simple integrations where you do not need business logic — just pass through to an AWS service.

Stages and Deployments

Changes to your API do not go live automatically. You must deploy them to a stage.

◈ DIAGRAM

Your API definition (routes, integrations, auth)
    ↓ Deploy
Stage: dev   → https://abc123.execute-api.ap-south-1.amazonaws.com/dev/orders
Stage: prod  → https://abc123.execute-api.ap-south-1.amazonaws.com/prod/orders

Each stage can have different settings — throttling limits, caching, logging, and stage variables. Stage variables work like environment variables for your API — point dev stage at a dev Lambda and prod stage at a prod Lambda.

Canary deployments:

Route a percentage of prod traffic to a new version before fully deploying.

◈ DIAGRAM

prod stage → 95% to current Lambda version
           → 5% to new Lambda version (canary)
Monitor errors → if clean, promote canary to 100%
If errors → rollback instantly

API Gateway Security

IAM Authentication:

Sign requests with AWS Signature Version 4. Correct for AWS service-to-service calls where the caller has an IAM role. Good for internal APIs.

Lambda Authorizer (formerly Custom Authorizer):

A Lambda function that runs before your backend. It receives the request token, validates it (JWT, OAuth, API key, anything), and returns an IAM policy — Allow or Deny.

◈ DIAGRAM

Client sends request with Bearer token
    ↓
API Gateway calls your Lambda Authorizer
    ↓
Authorizer validates token against your auth server
    ↓
Returns: Allow → request passes to backend
Returns: Deny  → 403 returned immediately

Result is cached for up to 3600 seconds so you do not validate the same token on every request.

Cognito User Pools:

The simplest option. Users log in through Cognito, get a JWT token. They include the token in API requests. API Gateway validates the token directly — no authorizer Lambda needed.

◈ DIAGRAM

User logs in → Cognito issues JWT token
User sends API request with Authorization: Bearer JWT
API Gateway validates JWT against Cognito automatically
Valid → request passes   Invalid → 401 Unauthorized

WebSocket APIs

HTTP APIs are one direction — client sends, server responds. WebSocket APIs keep the connection open in both directions.

◈ DIAGRAM

Client connects → WebSocket connection established (stays open)
Server pushes data to client at any time without client asking
Client sends messages to server at any time

Use cases:

Real-time chat (WhatsApp-style messaging)
Live dashboards (stock prices, cricket scores updating automatically)
Multiplayer gaming
Live order tracking (Swiggy driver location updating in real time)

API Gateway manages the connections. Your Lambda handles the messages. When Lambda wants to push to a client it calls the API Gateway Management API with the connection ID.

AWS Step Functions — Orchestrating Workflows

At Swiggy, placing an order triggers six things in sequence — fraud check, then inventory check, then kitchen notification, then delivery assignment, then email confirmation, then analytics update. Each step depends on the previous.

Without Step Functions you write glue code — each service calls the next, handles failures, retries, and timeouts. When something fails halfway through you have no visibility into which step failed.

With Step Functions you define the workflow visually. Each step is a state. Step Functions manages the execution, retries, error handling, and gives you a full execution history.

◈ DIAGRAM

Order placed
    ↓ State: FraudCheck
    ↓ State: InventoryReserve
    ↓ State: KitchenNotify
    ↓ State: DeliveryAssign (parallel with KitchenNotify)
    ↓ State: EmailConfirm
    ↓ State: Analytics
    ↓ Done

Standard vs Express Workflows:

	Standard	Express
Duration	Up to 1 year	Up to 5 minutes
Execution model	Exactly-once	At-least-once
Execution history	Full visible history	CloudWatch Logs only
Pricing	Per state transition	Per execution duration
Best for	Long-running business processes	High-volume short tasks

Standard workflow example: order fulfilment that may pause for hours waiting for a warehouse to confirm stock.

Express workflow example: processing 1 million IoT events per day where each pipeline takes 30 seconds.

Error handling built in:

◈ DIAGRAM

State fails → Retry (up to N times with backoff)
Still failing → Catch → route to error handling state
No manual try/catch code in your Lambda functions

Amazon Cognito — User Identity

Cognito handles user sign-up, sign-in, and access control for your applications. Two completely separate services under one name.

User Pools — who you are:

A user directory. Stores usernames, passwords, email, phone, and custom attributes. Handles sign-up, login, MFA, password reset, and email verification.

◈ DIAGRAM

User signs up → Cognito User Pool stores their account
User logs in → Cognito validates credentials → returns JWT tokens:
  ID Token     → who the user is (name, email, custom claims)
  Access Token → what they are allowed to do
  Refresh Token → get new tokens without re-login

User Pools integrate directly with API Gateway — no code needed to validate tokens.

Social login support: Google, Facebook, Apple, SAML identity providers. Cognito handles the OAuth dance. Your app just receives a Cognito JWT.

Identity Pools — what you can do in AWS:

Identity Pools give users temporary AWS credentials so they can call AWS services directly from the app — upload a file to S3, read from DynamoDB, call an API.

◈ DIAGRAM

User logged in via Cognito User Pool
    ↓
App calls Identity Pool: "exchange this JWT for AWS credentials"
    ↓
Identity Pool returns: temporary Access Key + Secret + Session Token
    ↓
App calls S3 directly using those credentials

Row-level DynamoDB security with Identity Pools:

Each user only sees their own data — enforced at the AWS level, not the application level.

TEXT

IAM policy condition:
dynamodb:LeadingKeys must match ${cognito-identity.amazonaws.com:sub}
 
User A's identity sub: abc-123
User A can only read DynamoDB items where primary key = abc-123
User A cannot read User B's items even if they try

User Pools vs Identity Pools:

	User Pools	Identity Pools
Purpose	User authentication	AWS service access
Issues	JWT tokens	Temporary AWS credentials
Think of it as	Your login system	Your AWS permissions system
Use for	Sign in, sign up, MFA	Upload to S3, call DynamoDB from app

Remember
User Pools = authentication (who are you). Identity Pools = authorisation (what AWS resources can you access). They are used together. User Pool handles login. Identity Pool exchanges the User Pool JWT for temporary AWS credentials.

Hands-on Lab — API Gateway with Lambda and Cognito Auth

Step 1 — Create the Lambda function

◈ DIAGRAM

Lambda → Create function
Name: devops-orders-api
Runtime: Python 3.12
Create function
 
Replace code with:

Python

import json
 
def lambda_handler(event, context):
    method = event.get('httpMethod', 'GET')
    path = event.get('path', '/')
 
    if method == 'GET' and '/orders' in path:
        return {
            'statusCode': 200,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps([
                {'orderId': 'ORD-001', 'city': 'Mumbai', 'amount': 450},
                {'orderId': 'ORD-002', 'city': 'Pune', 'amount': 1200}
            ])
        }
 
    return {'statusCode': 404, 'body': json.dumps({'error': 'Not found'})}

TEXT

Deploy

Step 2 — Create the REST API

◈ DIAGRAM

API Gateway → APIs → Create API
REST API → Build
API name: devops-orders-api
Endpoint type: Regional
Create API

Step 3 — Create resource and method

◈ DIAGRAM

Resources → Actions → Create Resource
Resource name: orders    Resource path: /orders    Create Resource
 
Select /orders → Actions → Create Method → GET → tick
Integration type: Lambda Function
Lambda function: devops-orders-api
Save → OK (grant permission)

Step 4 — Deploy the API

◈ DIAGRAM

Actions → Deploy API
Deployment stage: [New Stage]    Stage name: dev
Deploy
 
Copy the Invoke URL:
https://abc123.execute-api.ap-south-1.amazonaws.com/dev
 
Test in browser:
https://abc123.execute-api.ap-south-1.amazonaws.com/dev/orders
Returns the JSON order list from your Lambda.

Step 5 — Create a Cognito User Pool

◈ DIAGRAM

Cognito → User pools → Create user pool
Sign-in options: Email
Password policy: Cognito defaults
MFA: No MFA (for lab)
User pool name: devops-users
App client name: devops-web-app
Create user pool
 
Note the User Pool ID and App Client ID.

Step 6 — Add Cognito authorizer to API

◈ DIAGRAM

API Gateway → devops-orders-api → Authorizers → Create New Authorizer
Name: cognito-auth
Type: Cognito
Cognito User Pool: devops-users
Token source: Authorization
Create
 
API Gateway → Resources → /orders → GET → Method Request
Authorization → select cognito-auth → tick
 
Re-deploy: Actions → Deploy API → dev stage

Step 7 — Create a test user and get token

◈ DIAGRAM

Cognito → devops-users → Users → Create user
Email: test@devops.in    Temporary password: Test@1234
Create user

Bash

## Get auth token via CLI
aws cognito-idp initiate-auth \
  --auth-flow USER_PASSWORD_AUTH \
  --client-id YOUR-APP-CLIENT-ID \
  --auth-parameters USERNAME=test@devops.in,PASSWORD=Test@1234 \
  --region ap-south-1
## Copy the IdToken from the response

Bash

Test with token:
curl -H "Authorization: PASTE-ID-TOKEN-HERE" \
https://abc123.execute-api.ap-south-1.amazonaws.com/dev/orders
Returns orders (authenticated)
 
Test without token:
curl https://abc123.execute-api.ap-south-1.amazonaws.com/dev/orders
Returns 401 Unauthorized

Step 8 — Cleanup

◈ DIAGRAM

API Gateway → devops-orders-api → Delete API
Lambda → devops-orders-api → Delete function
Cognito → devops-users → Delete user pool

Common Mistakes to Avoid

Common Mistake
Using Edge-Optimized endpoint with an SSL certificate in the wrong region. Edge-Optimized uses CloudFront, and all CloudFront certificates must be in us-east-1. If your cert is in ap-south-1 and your endpoint is Edge-Optimized, HTTPS will not work. Either use Regional endpoint with the cert in the same region, or move the cert to us-east-1.

Common Mistake
Confusing Cognito User Pools with Identity Pools. User Pools handle user login — who you are. Identity Pools give users temporary AWS credentials — what AWS services they can access. They solve different problems. Most apps need both working together.

Common Mistake
Choosing Standard Step Functions workflow for high-volume short tasks. Standard workflows charge per state transition and store full execution history. At 1 million executions per day this becomes expensive. Use Express workflows for high-volume, short-duration pipelines.

Tip
Use API Gateway's built-in usage plans and API keys to implement rate limiting per client. Each API key gets a throttle limit and a quota. Clients that exceed their limit get 429 Too Many Requests automatically — no code in your Lambda needed.