Build Production AWS Infrastructure with Terraform

Provision a production-style AWS stack with Terraform: VPC, EKS, IRSA, ECR and remote state, then deploy Capstone 1's app on it.

Architecture Overview & Problem Statement

This project builds a production-style AWS foundation for Kubernetes with Terraform. You provision a VPC with public and private subnets, an EKS cluster whose nodes run in private subnets, an ECR registry, IRSA roles for pod-level AWS access, and an S3 backend with locking for Terraform state, then deploy Capstone 1's application onto it.

In Capstone 1 you built and deployed a three-tier application to Kubernetes. But there was a gap: where did that cluster come from? Who designed the network? Who configured the cloud permissions?

In most learning environments you are handed a pre-built cluster. In a real company, a Platform Engineer provisions it. They decide the network layout, set up the permissions, configure the container registry, and make sure everything is secure and reproducible.

This capstone fills that gap. You provision real AWS infrastructure from scratch with Terraform, the way teams at companies like Razorpay, Zerodha and Hotstar set up their environments.

What You Build

flowchart TD S[(Remote state
S3 + DynamoDB)] --> T[Terraform] T --> V[VPC and
subnets] T --> E[EKS cluster] T --> R[ECR registry] classDef blue fill:#326CE5,stroke:#1E4FB8,color:#fff classDef green fill:#10B981,stroke:#0B7A56,color:#fff classDef orange fill:#F97316,stroke:#B85410,color:#fff classDef red fill:#EF4444,stroke:#B02D2D,color:#fff classDef purple fill:#8B5CF6,stroke:#6238C2,color:#fff classDef slate fill:#475569,stroke:#2B3544,color:#fff class S slate class T purple class V blue class E blue class R orange

Once it is running, images and AWS access flow like this:

flowchart LR R[ECR
image] -->|pulled by| N[EKS
node] N --> P[Backend
pod] P -->|IRSA role| B[(S3
bucket)] classDef blue fill:#326CE5,stroke:#1E4FB8,color:#fff classDef green fill:#10B981,stroke:#0B7A56,color:#fff classDef orange fill:#F97316,stroke:#B85410,color:#fff classDef red fill:#EF4444,stroke:#B02D2D,color:#fff classDef purple fill:#8B5CF6,stroke:#6238C2,color:#fff classDef slate fill:#475569,stroke:#2B3544,color:#fff class R orange class N blue class P green class B slate

The cluster also gets the EBS CSI driver add-on, so the persistent volumes from Capstone 1 work.

After this capstone you can:

Explain how VPCs, subnets and routing fit together in AWS
Provision an EKS cluster with Terraform using reusable modules
Explain why IRSA beats access keys, and prove it works from inside a pod
Manage Terraform state safely with an S3 backend
Recover from broken state, import existing resources, and move resources between modules, in a staging environment where mistakes are cheap
Deploy Capstone 1's application onto infrastructure you built yourself

Portfolio deliverable: a GitHub repo platform-infrastructure with the modules, a production and a staging environment, an architecture diagram, the output of your aws sts get-caller-identity run from inside a pod (account ID redacted) proving IRSA works, and a short "what I would do differently in a real company" section.

Time to complete: 3-4 hours, much of it waiting for AWS to create the cluster.

Tip
This project creates billable resources. The EKS control plane, the NAT gateways, and the EC2 nodes all charge by the hour, so the biggest cost is leaving it running. Check the current AWS pricing pages for EKS, NAT Gateway and EC2 in ap-south-1 before you start. Complete the Cleanup section the same day. The staging environment uses a single NAT gateway to keep the cost lower.

What you need before starting:

An AWS account you can safely experiment in (not a shared production account)
AWS CLI v2 configured (aws configure)
Terraform 1.5 or newer
kubectl, Docker, and jq installed
Capstone 1's code, with its k8s/ manifests, available locally

Bash

## Verify tools are installed
aws --version
terraform --version
kubectl version --client
docker --version
jq --version
 
## Verify AWS credentials work
aws sts get-caller-identity
## Should show your Account ID, User ID and ARN
## If this fails, run: aws configure

What Is Terraform and Why Not Just Use the AWS Console?

You could click through the AWS Console and create everything by hand. Many people do this while learning. The problems:

It is not reproducible. Three months later you need a staging environment that matches production. You have to remember every click and every setting. You will miss something, and the environments will drift apart.

It is not reviewable. When a colleague changes a security group in the console, nobody else knows. There is no pull request, no code review, no history.

It is not recoverable. If someone deletes the cluster, recreating it from memory is slow, error-prone and stressful.

Terraform solves this. You describe infrastructure in .tf files. Terraform reads them and creates, updates or destroys real AWS resources to match. The files live in Git, so every change is tracked, reviewed and recoverable.

Key concepts before you start:

Provider is a plugin that knows how to talk to a cloud or service. We use the AWS provider (and the TLS provider, to read a certificate).

Resource is one piece of infrastructure. An aws_vpc is a resource. An aws_eks_cluster is a resource.

Module is a reusable group of resources. Instead of writing 100 lines for a VPC every time, you call a module with 10 lines.

State is Terraform's record of what it created. It is how Terraform knows what exists and what needs to change.

Plan shows what Terraform will do before it does it. Always read the plan.

Apply actually creates or changes the infrastructure.

Milestone 1: Set Up the Project and Remote State

This milestone creates the project layout and the S3 bucket and DynamoDB table that will hold Terraform state.

Why Remote State?

By default Terraform stores state in a local file called terraform.tfstate. That works for learning but breaks in a team: two engineers applying at once can corrupt the state, and if your laptop dies the state is gone and Terraform no longer knows what it created.

Remote state keeps the file in S3 (durable, versioned, encrypted) and uses a lock so only one person can apply at a time.

We create the S3 bucket and the lock table first, by hand, once. Terraform cannot manage the backend that stores its own state.

1.1 Project Structure

Bash

mkdir platform-infrastructure && cd platform-infrastructure
git init
 
mkdir -p modules/vpc modules/eks modules/ecr
mkdir -p environments/production environments/staging
 
cat > .gitignore << 'EOF'
.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
EOF
 
echo "Project structure created"

◈ DIAGRAM

platform-infrastructure/
  modules/
    vpc/         <- reusable VPC module
    eks/         <- reusable EKS module (cluster, nodes, OIDC, EBS CSI)
    ecr/         <- reusable ECR module
  environments/
    production/  <- production configuration
    staging/     <- staging configuration (same modules, smaller values)

This separates what (modules, the reusable logic) from where (environments, the specific values). The same VPC module builds a large production network and a small staging one.

Tip
Commit .terraform.lock.hcl (created by terraform init) so everyone uses the same provider versions. Never commit state files or plan files; they can contain secrets.

1.2 Create the Remote State Infrastructure

Bash

## Set your AWS region
export AWS_REGION=ap-south-1    ## Mumbai
 
## Bucket names must be globally unique - include your account ID
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export TF_STATE_BUCKET="terraform-state-${ACCOUNT_ID}-${AWS_REGION}"
 
aws s3api create-bucket \
  --bucket ${TF_STATE_BUCKET} \
  --region ${AWS_REGION} \
  --create-bucket-configuration LocationConstraint=${AWS_REGION}
 
## Versioning lets you recover earlier state files
aws s3api put-bucket-versioning \
  --bucket ${TF_STATE_BUCKET} \
  --versioning-configuration Status=Enabled
 
## State files can contain sensitive values - encrypt them
aws s3api put-bucket-encryption \
  --bucket ${TF_STATE_BUCKET} \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "AES256" }
    }]
  }'
 
## State must NEVER be public
aws s3api put-public-access-block \
  --bucket ${TF_STATE_BUCKET} \
  --public-access-block-configuration \
  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
 
## DynamoDB table used for state locking
aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region ${AWS_REGION}
 
echo "S3 Bucket: ${TF_STATE_BUCKET}"
echo "DynamoDB Table: terraform-state-lock"

Tip
DynamoDB-based locking is what this course uses. Newer Terraform versions also support locking directly in S3 (the use_lockfile option), and DynamoDB locking is being phased out. If your Terraform version warns about it, check the S3 backend documentation for your version. The ideas (one writer at a time) are identical.

Common Mistake
Committing terraform.tfstate or .tfplan files to Git. They can contain secrets. Keep them out with the .gitignore above and use the remote backend.

Milestone 2: Build the VPC Module

This milestone builds a reusable Terraform VPC module with public subnets, private subnets, NAT gateways and route tables.

What Is a VPC and Why Do We Need One?

A VPC (Virtual Private Cloud) is your private network inside AWS. Think of it as renting an isolated section of the AWS data centre. Nothing gets in or out unless you allow it.

Inside the VPC we create subnets:

Public subnets are connected to the internet through an Internet Gateway. Load balancers live here.

Private subnets are not directly reachable from the internet. EKS worker nodes live here. They reach the internet through a NAT Gateway (to pull packages and call AWS APIs), but the internet cannot start connections to them.

Public subnets for load balancers and private subnets for compute is the standard pattern for secure AWS architectures.

flowchart TD I[Internet] --> G[Internet
Gateway] G --> U[Public subnets
load balancers] U --> N[NAT gateway] N --> V[Private subnets
EKS nodes] classDef blue fill:#326CE5,stroke:#1E4FB8,color:#fff classDef green fill:#10B981,stroke:#0B7A56,color:#fff classDef orange fill:#F97316,stroke:#B85410,color:#fff classDef red fill:#EF4444,stroke:#B02D2D,color:#fff classDef purple fill:#8B5CF6,stroke:#6238C2,color:#fff classDef slate fill:#475569,stroke:#2B3544,color:#fff class I slate class G purple class U orange class N purple class V green

Public subnets use 10.0.0.0/24 and 10.0.1.0/24. Private subnets use 10.0.16.0/20 and 10.0.32.0/20.

Tip
A CIDR block must start on its own boundary. 10.0.16.0/20 is valid because 16 is a multiple of 16. 10.0.10.0/20 is not: AWS would reject it, because a /20 must start at 10.0.0.0, 10.0.16.0, 10.0.32.0, and so on. When a CIDR is rejected, check the boundary first.

2.1 VPC Module

HCL

# modules/vpc/main.tf
 
# ── Variables ─────────────────────────────────────────────────
# Variables are like function parameters - they make the module reusable
variable "cluster_name" {
  description = "Name of the EKS cluster - used for naming and tagging"
  type        = string
}
 
variable "environment" {
  description = "Environment name (production, staging)"
  type        = string
}
 
variable "vpc_cidr" {
  description = "IP address range for the VPC"
  type        = string
  default     = "10.0.0.0/16"
}
 
variable "availability_zones" {
  description = "AZs to create subnets in - use 2 or more for high availability"
  type        = list(string)
}
 
variable "public_subnet_cidrs" {
  description = "IP ranges for public subnets - one per AZ"
  type        = list(string)
}
 
variable "private_subnet_cidrs" {
  description = "IP ranges for private subnets - one per AZ"
  type        = list(string)
}
 
variable "single_nat_gateway" {
  description = "true = one shared NAT gateway (cheaper, one point of failure). false = one per AZ."
  type        = bool
  default     = false
}
 
locals {
  nat_count = var.single_nat_gateway ? 1 : length(var.availability_zones)
}
 
# ── VPC ───────────────────────────────────────────────────────
resource "aws_vpc" "main" {
  cidr_block = var.vpc_cidr
 
  # EKS needs DNS hostnames and DNS support inside the VPC
  enable_dns_hostnames = true
  enable_dns_support   = true
 
  tags = {
    Name        = "${var.cluster_name}-vpc"
    Environment = var.environment
  }
}
 
# ── Internet Gateway ──────────────────────────────────────────
# Connects the VPC to the public internet
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
 
  tags = {
    Name        = "${var.cluster_name}-igw"
    Environment = var.environment
  }
}
 
# ── Public Subnets ────────────────────────────────────────────
# count = length(...) creates one subnet per availability zone
resource "aws_subnet" "public" {
  count = length(var.availability_zones)
 
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.public_subnet_cidrs[count.index]
  availability_zone       = var.availability_zones[count.index]
  map_public_ip_on_launch = true
 
  tags = {
    Name        = "${var.cluster_name}-public-${var.availability_zones[count.index]}"
    Environment = var.environment
    # Tells Kubernetes this subnet is for internet-facing load balancers
    "kubernetes.io/role/elb"                    = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  }
}
 
# ── Private Subnets ───────────────────────────────────────────
resource "aws_subnet" "private" {
  count = length(var.availability_zones)
 
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.private_subnet_cidrs[count.index]
  availability_zone       = var.availability_zones[count.index]
  map_public_ip_on_launch = false
 
  tags = {
    Name        = "${var.cluster_name}-private-${var.availability_zones[count.index]}"
    Environment = var.environment
    # Tells Kubernetes this subnet is for internal load balancers
    "kubernetes.io/role/internal-elb"           = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  }
}
 
# ── Elastic IPs for NAT Gateways ──────────────────────────────
# A NAT Gateway needs a fixed public IP address
resource "aws_eip" "nat" {
  count  = local.nat_count
  domain = "vpc"
 
  tags = {
    Name        = "${var.cluster_name}-nat-eip-${count.index}"
    Environment = var.environment
  }
 
  depends_on = [aws_internet_gateway.main]
}
 
# ── NAT Gateways ──────────────────────────────────────────────
# Let private subnets reach the internet without being reachable from it.
# NAT Gateways live in PUBLIC subnets.
resource "aws_nat_gateway" "main" {
  count = local.nat_count
 
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id
 
  tags = {
    Name        = "${var.cluster_name}-nat-${count.index}"
    Environment = var.environment
  }
 
  depends_on = [aws_internet_gateway.main]
}
 
# ── Route Tables ──────────────────────────────────────────────
# A route table tells traffic where to go.
# Public: anything not inside the VPC goes to the Internet Gateway.
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
 
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }
 
  tags = {
    Name        = "${var.cluster_name}-public-rt"
    Environment = var.environment
  }
}
 
resource "aws_route_table_association" "public" {
  count = length(var.availability_zones)
 
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}
 
# Private: one route table per AZ, pointing at a NAT Gateway
resource "aws_route_table" "private" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.main.id
 
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main[var.single_nat_gateway ? 0 : count.index].id
  }
 
  tags = {
    Name        = "${var.cluster_name}-private-rt-${var.availability_zones[count.index]}"
    Environment = var.environment
  }
}
 
resource "aws_route_table_association" "private" {
  count = length(var.availability_zones)
 
  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private[count.index].id
}
 
# ── Outputs ───────────────────────────────────────────────────
# Outputs expose values so other modules can use them
output "vpc_id" {
  description = "VPC ID"
  value       = aws_vpc.main.id
}
 
output "public_subnet_ids" {
  description = "IDs of the public subnets"
  value       = aws_subnet.public[*].id
}
 
output "private_subnet_ids" {
  description = "IDs of the private subnets - EKS nodes go here"
  value       = aws_subnet.private[*].id
}

Common Mistake
Choosing a CIDR that is not on its own boundary, or two subnets that overlap. AWS rejects it at apply time, after other resources may already exist. Check boundaries and overlaps before you apply.

Milestone 3: Build the EKS Module

This milestone builds the EKS module: the cluster, a managed node group in private subnets, an OIDC provider for IRSA, and the EBS CSI driver.

What Is EKS and What Does AWS Manage for You?

EKS (Elastic Kubernetes Service) is AWS's managed Kubernetes. AWS runs the control plane for you.

AWS manages:

The Kubernetes API server (availability, patching)
etcd, the database where Kubernetes stores its state
The control plane nodes

You manage:

Worker nodes (the EC2 instances where your pods run)
Networking configuration
IAM permissions

What Is IRSA and Why Does It Matter?

IRSA (IAM Roles for Service Accounts) is the standard way to give pods permission to call AWS services.

The wrong way, which many people do first:

◈ DIAGRAM

Put AWS access keys in environment variables or a Kubernetes Secret
-> Keys are easy to leak into Git or logs
-> Every pod using the Secret shares the same permissions
-> Keys are long-lived and rarely rotated

The right way (IRSA):

◈ DIAGRAM

An IAM role is linked to a Kubernetes Service Account
-> No long-lived keys anywhere
-> Each workload gets only the permissions it needs
-> AWS issues temporary credentials automatically

IRSA works by linking an IAM role to a Service Account through OIDC (OpenID Connect), a standard for federated identity. IAM trusts tokens issued by your cluster, and the role's trust policy names the exact namespace and Service Account allowed to use it.

Tip
AWS also offers a newer mechanism called EKS Pod Identity. IRSA is still widely used and is what you will meet in most existing clusters, so it is worth learning first.

3.1 IAM Roles for EKS

HCL

# modules/eks/iam.tf
 
# ── EKS Cluster Role ──────────────────────────────────────────
# Used by the EKS control plane to manage AWS resources on your behalf
 
data "aws_iam_policy_document" "eks_cluster_assume_role" {
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["eks.amazonaws.com"]
    }
    actions = ["sts:AssumeRole"]
  }
}
 
resource "aws_iam_role" "eks_cluster" {
  name               = "${var.cluster_name}-cluster-role"
  assume_role_policy = data.aws_iam_policy_document.eks_cluster_assume_role.json
 
  tags = {
    Name        = "${var.cluster_name}-cluster-role"
    Environment = var.environment
  }
}
 
resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster.name
}
 
# ── EKS Node Role ─────────────────────────────────────────────
# Used by the EC2 worker nodes: join the cluster, set up pod networking,
# and pull images from ECR
 
data "aws_iam_policy_document" "eks_node_assume_role" {
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
    actions = ["sts:AssumeRole"]
  }
}
 
resource "aws_iam_role" "eks_node" {
  name               = "${var.cluster_name}-node-role"
  assume_role_policy = data.aws_iam_policy_document.eks_node_assume_role.json
 
  tags = {
    Name        = "${var.cluster_name}-node-role"
    Environment = var.environment
  }
}
 
# Three policies are required for managed node groups
resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node.name
}
 
resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  # The CNI plugin manages pod networking and needs VPC permissions
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node.name
}
 
resource "aws_iam_role_policy_attachment" "ecr_read_only" {
  # Nodes pull images from ECR using this role - no passwords needed
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node.name
}

3.2 EKS Cluster, Node Group and EBS CSI Driver

HCL

# modules/eks/main.tf
 
variable "cluster_name" {
  type = string
}
 
variable "cluster_version" {
  description = "Kubernetes version. Check the EKS docs for the newest version in standard support."
  type        = string
  default     = "1.34"
}
 
variable "environment" {
  type = string
}
 
variable "private_subnet_ids" {
  type = list(string)
}
 
variable "public_subnet_ids" {
  type = list(string)
}
 
variable "public_access_cidrs" {
  description = "Who may reach the Kubernetes API from the internet. Restrict this to your own IP in real use."
  type        = list(string)
  default     = ["0.0.0.0/0"]
}
 
variable "node_instance_types" {
  type    = list(string)
  default = ["t3.medium"]
}
 
variable "node_desired_size" {
  type    = number
  default = 2
}
 
variable "node_min_size" {
  type    = number
  default = 1
}
 
variable "node_max_size" {
  type    = number
  default = 5
}
 
# ── EKS Cluster ───────────────────────────────────────────────
resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  version  = var.cluster_version
  role_arn = aws_iam_role.eks_cluster.arn
 
  vpc_config {
    subnet_ids = concat(var.private_subnet_ids, var.public_subnet_ids)
 
    # kubectl from inside the VPC
    endpoint_private_access = true
 
    # kubectl from your laptop. In production use a VPN or bastion and set this to false,
    # or at least restrict public_access_cidrs to known IP ranges.
    endpoint_public_access = true
    public_access_cidrs    = var.public_access_cidrs
  }
 
  # Control plane logs to CloudWatch - useful for debugging auth and API issues
  enabled_cluster_log_types = ["api", "audit", "authenticator"]
 
  tags = {
    Name        = var.cluster_name
    Environment = var.environment
  }
 
  depends_on = [aws_iam_role_policy_attachment.eks_cluster_policy]
}
 
# ── OIDC Provider for IRSA ────────────────────────────────────
# Creates the trust relationship between this cluster and AWS IAM, so IAM
# can verify that a token really came from a pod in THIS cluster.
 
data "tls_certificate" "eks" {
  url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}
 
resource "aws_iam_openid_connect_provider" "eks" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.main.identity[0].oidc[0].issuer
 
  tags = {
    Name        = "${var.cluster_name}-oidc"
    Environment = var.environment
  }
}
 
locals {
  # IRSA trust policies use the issuer WITHOUT the https:// prefix
  oidc_issuer = replace(aws_eks_cluster.main.identity[0].oidc[0].issuer, "https://", "")
}
 
# ── Node Group ────────────────────────────────────────────────
# AWS creates, patches and replaces the EC2 instances for you
resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "${var.cluster_name}-nodes"
  node_role_arn   = aws_iam_role.eks_node.arn
 
  # Nodes go in private subnets only
  subnet_ids     = var.private_subnet_ids
  instance_types = var.node_instance_types
  capacity_type  = "ON_DEMAND"
 
  scaling_config {
    desired_size = var.node_desired_size
    min_size     = var.node_min_size
    max_size     = var.node_max_size
  }
 
  # Replace nodes one at a time during updates
  update_config {
    max_unavailable = 1
  }
 
  # If a cluster autoscaler changes the node count later,
  # do not let the next terraform apply fight it
  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
 
  tags = {
    Name        = "${var.cluster_name}-node-group"
    Environment = var.environment
  }
 
  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ecr_read_only,
  ]
}
 
# ── EBS CSI Driver (add-on) ───────────────────────────────────
# Without this driver, a PersistentVolumeClaim (like Capstone 1's database volume)
# can stay Pending forever. The driver runs as a pod in kube-system and needs AWS
# permissions - so it is our first real IRSA role.
 
data "aws_iam_policy_document" "ebs_csi_assume_role" {
  statement {
    effect = "Allow"
 
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.eks.arn]
    }
 
    actions = ["sts:AssumeRoleWithWebIdentity"]
 
    condition {
      test     = "StringEquals"
      variable = "${local.oidc_issuer}:sub"
      values   = ["system:serviceaccount:kube-system:ebs-csi-controller-sa"]
    }
 
    condition {
      test     = "StringEquals"
      variable = "${local.oidc_issuer}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}
 
resource "aws_iam_role" "ebs_csi" {
  name               = "${var.cluster_name}-ebs-csi"
  assume_role_policy = data.aws_iam_policy_document.ebs_csi_assume_role.json
}
 
resource "aws_iam_role_policy_attachment" "ebs_csi" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
  role       = aws_iam_role.ebs_csi.name
}
 
resource "aws_eks_addon" "ebs_csi" {
  cluster_name             = aws_eks_cluster.main.name
  addon_name               = "aws-ebs-csi-driver"
  service_account_role_arn = aws_iam_role.ebs_csi.arn
 
  depends_on = [aws_eks_node_group.main]
}
 
# ── Outputs ───────────────────────────────────────────────────
output "cluster_name" {
  value = aws_eks_cluster.main.name
}
 
output "cluster_endpoint" {
  value = aws_eks_cluster.main.endpoint
}
 
output "oidc_provider_arn" {
  value = aws_iam_openid_connect_provider.eks.arn
}
 
output "oidc_issuer" {
  description = "OIDC issuer without https:// - use it in IRSA trust policies"
  value       = local.oidc_issuer
}

Tip
The IAM identity that runs terraform apply is the one that gets admin access to the new cluster by default. If a teammate runs kubectl and gets "Unauthorized", the cluster is working as designed. You must grant their IAM identity access (through EKS access entries). This catches many people out on their first shared cluster.

Common Mistake
Putting https:// in the IRSA trust-policy condition key. The key must use the issuer without the scheme, which is why this module exposes oidc_issuer.

Milestone 4: Build the ECR Module

This milestone builds an ECR module that creates private image repositories with scanning and lifecycle rules.

What Is ECR and Why Not Use Docker Hub?

ECR (Elastic Container Registry) is AWS's private container registry.

Why not Docker Hub?

Docker Hub applies pull-rate limits to free and anonymous users
ECR is inside AWS, so pulls to your cluster are fast
ECR uses IAM: your EKS nodes pull images with their role, no passwords stored anywhere
ECR can scan images for known vulnerabilities on push

HCL

# modules/ecr/main.tf
 
variable "repositories" {
  description = "List of repository names to create"
  type        = list(string)
}
 
variable "environment" {
  type = string
}
 
resource "aws_ecr_repository" "main" {
  for_each = toset(var.repositories)
 
  name = each.value
 
  # IMMUTABLE: a tag like v1.0.0 can never be overwritten, so what you tested
  # is exactly what runs. Use unique tags (version or git SHA) for every push.
  image_tag_mutability = "IMMUTABLE"
 
  image_scanning_configuration {
    scan_on_push = true
  }
 
  encryption_configuration {
    encryption_type = "AES256"
  }
 
  tags = {
    Name        = each.value
    Environment = var.environment
  }
}
 
# Lifecycle policy: delete untagged images after 1 day,
# keep only the newest 30 images whose tag starts with "v"
resource "aws_ecr_lifecycle_policy" "main" {
  for_each   = toset(var.repositories)
  repository = aws_ecr_repository.main[each.value].name
 
  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Remove untagged images after 1 day"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 1
        }
        action = { type = "expire" }
      },
      {
        rulePriority = 2
        description  = "Keep only the last 30 version-tagged images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["v"]
          countType     = "imageCountMoreThan"
          countNumber   = 30
        }
        action = { type = "expire" }
      }
    ]
  })
}
 
output "repository_urls" {
  description = "Map of repository name to URL"
  value       = { for k, v in aws_ecr_repository.main : k => v.repository_url }
}

Common Mistake
Pushing the same tag twice to an IMMUTABLE repository. The second push is rejected. Use a new tag (a version or Git SHA) for every build.

Milestone 5: Configure the Production Environment

This milestone wires the modules into a production environment and adds a scoped IRSA role for the backend pods.

Connecting Everything Together

The environment calls each module with production values. It also creates an S3 bucket for the app and an IRSA role for your backend pods that can only touch that one bucket.

HCL

# environments/production/main.tf
 
terraform {
  required_version = ">= 1.5.0"
 
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }
 
  # Remote state. Replace YOUR_ACCOUNT_ID with the value from Milestone 1.
  backend "s3" {
    bucket         = "terraform-state-YOUR_ACCOUNT_ID-ap-south-1"
    key            = "production/terraform.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}
 
provider "aws" {
  region = "ap-south-1"
 
  default_tags {
    tags = {
      ManagedBy   = "terraform"
      Environment = "production"
      Project     = "platform-engineering"
    }
  }
}
 
data "aws_caller_identity" "current" {}
 
locals {
  cluster_name       = "platform-production"
  environment        = "production"
  region             = "ap-south-1"
  availability_zones = ["ap-south-1a", "ap-south-1b"]
}
 
# ── VPC ───────────────────────────────────────────────────────
module "vpc" {
  source = "../../modules/vpc"
 
  cluster_name       = local.cluster_name
  environment        = local.environment
  vpc_cidr           = "10.0.0.0/16"
  availability_zones = local.availability_zones
 
  # /24 = 254 usable IPs per public subnet
  public_subnet_cidrs = ["10.0.0.0/24", "10.0.1.0/24"]
 
  # /20 = about 4,000 IPs per private subnet (pods get VPC IPs on EKS).
  # Each /20 must start on a 16 boundary: .16.0 and .32.0 are valid.
  private_subnet_cidrs = ["10.0.16.0/20", "10.0.32.0/20"]
 
  # One NAT gateway per AZ: more resilient, more expensive
  single_nat_gateway = false
}
 
# ── EKS ───────────────────────────────────────────────────────
module "eks" {
  source = "../../modules/eks"
 
  cluster_name       = local.cluster_name
  cluster_version    = "1.34"
  environment        = local.environment
  private_subnet_ids = module.vpc.private_subnet_ids
  public_subnet_ids  = module.vpc.public_subnet_ids
 
  node_instance_types = ["t3.medium"]
  node_desired_size   = 2
  node_min_size       = 1
  node_max_size       = 4
}
 
# ── ECR ───────────────────────────────────────────────────────
module "ecr" {
  source = "../../modules/ecr"
 
  environment  = local.environment
  repositories = ["swiggy-backend", "swiggy-frontend"]
}
 
# ── S3 bucket for application assets ──────────────────────────
resource "aws_s3_bucket" "app_assets" {
  bucket = "swiggy-clone-assets-${data.aws_caller_identity.current.account_id}-${local.region}"
}
 
resource "aws_s3_bucket_public_access_block" "app_assets" {
  bucket                  = aws_s3_bucket.app_assets.id
  block_public_acls       = true
  ignore_public_acls      = true
  block_public_policy     = true
  restrict_public_buckets = true
}
 
# ── IRSA: backend pods may use ONLY the assets bucket ─────────
# Trust policy: only the 'backend-sa' Service Account in the 'swiggy-clone'
# namespace of THIS cluster can assume the role.
data "aws_iam_policy_document" "backend_assume_role" {
  statement {
    effect = "Allow"
 
    principals {
      type        = "Federated"
      identifiers = [module.eks.oidc_provider_arn]
    }
 
    actions = ["sts:AssumeRoleWithWebIdentity"]
 
    condition {
      test     = "StringEquals"
      variable = "${module.eks.oidc_issuer}:sub"
      values   = ["system:serviceaccount:swiggy-clone:backend-sa"]
    }
 
    condition {
      test     = "StringEquals"
      variable = "${module.eks.oidc_issuer}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}
 
resource "aws_iam_role" "backend" {
  name               = "${local.cluster_name}-backend-s3"
  assume_role_policy = data.aws_iam_policy_document.backend_assume_role.json
}
 
data "aws_iam_policy_document" "backend_s3" {
  statement {
    effect    = "Allow"
    actions   = ["s3:GetObject", "s3:PutObject"]
    resources = ["${aws_s3_bucket.app_assets.arn}/*"]
  }
 
  statement {
    effect    = "Allow"
    actions   = ["s3:ListBucket"]
    resources = [aws_s3_bucket.app_assets.arn]
  }
}
 
resource "aws_iam_role_policy" "backend_s3" {
  name   = "assets-bucket-access"
  role   = aws_iam_role.backend.id
  policy = data.aws_iam_policy_document.backend_s3.json
}

HCL

# environments/production/outputs.tf
 
output "cluster_name" {
  description = "Use with: aws eks update-kubeconfig --name <value>"
  value       = module.eks.cluster_name
}
 
output "cluster_endpoint" {
  description = "Kubernetes API server endpoint"
  value       = module.eks.cluster_endpoint
}
 
output "ecr_repository_urls" {
  description = "Map of ECR repository name to URL"
  value       = module.ecr.repository_urls
}
 
output "assets_bucket" {
  value = aws_s3_bucket.app_assets.bucket
}
 
output "backend_irsa_role_arn" {
  description = "Put this in the backend ServiceAccount annotation"
  value       = aws_iam_role.backend.arn
}

Security
Notice what the backend role is not: it is not AdministratorAccess, and it is not tied to ECR. Pods do not need a role to pull their own images, because the node role does that. IRSA is for what your application code calls: here, one bucket. When you add a new AWS dependency, add a new narrow statement. Never widen this one into *.

Common Mistake
Forgetting to replace YOUR_ACCOUNT_ID in the backend block. terraform init then fails with a bucket-not-found error.

Milestone 6: Deploy the Infrastructure

This milestone runs terraform init, plan and apply to create the infrastructure, then connects kubectl to the new cluster.

The Terraform Workflow

init downloads providers and modules and connects the backend
plan shows what will be created, changed or destroyed, without doing it
apply creates the infrastructure

The plan marks each resource: + create, ~ change in place, - destroy, -/+ destroy and recreate.

Command	What it does
`terraform init`	Downloads providers and modules, connects the S3 backend
`terraform fmt -recursive`	Formats all `.tf` files to the standard style
`terraform validate`	Checks syntax and references without calling AWS
`terraform plan -out=FILE`	Shows and saves the exact changes to be made
`terraform apply FILE`	Applies exactly the saved plan

Bash

cd environments/production
 
## Step 1: Initialise
terraform init
## Look for: "Successfully configured the backend "s3"!"
## and: "Terraform has been successfully initialized!"
 
## Step 2: Format and validate
terraform fmt -recursive
terraform validate
 
## Step 3: Plan and save the plan
terraform plan -out=production.tfplan
## Read it. For a fresh deployment, everything should be "to add".
## If anything shows "to destroy" that you did not expect - STOP and investigate.
 
## Step 4: Apply exactly the plan you reviewed
## EKS creation usually takes 10-15 minutes or more
terraform apply production.tfplan

Tip
terraform apply production.tfplan applies exactly what you reviewed. If you run a plain terraform apply an hour later, the plan can differ because the world changed. Saved plans are how real pipelines make "review, then apply" safe.

Connect kubectl to Your New Cluster

Bash

terraform output cluster_name
 
aws eks update-kubeconfig \
  --name platform-production \
  --region ap-south-1
 
kubectl get nodes
## NAME                                         STATUS   ROLES    AGE
## ip-10-0-xx-xx.ap-south-1.compute.internal    Ready    <none>   3m
## ip-10-0-xx-xx.ap-south-1.compute.internal    Ready    <none>   3m
 
kubectl get pods -n kube-system
## The ebs-csi-controller pods should be Running

Common Mistake
Running terraform apply without reading the plan. A -/+ line means a resource will be destroyed and recreated.

Milestone 7: Push Images and Deploy Capstone 1

This milestone pushes Capstone 1's images to ECR, deploys the app, and proves IRSA works from inside a pod.

Set two variables so the paths below work from any directory (adjust to where you keep the repos):

Bash

export INFRA_DIR=~/platform-infrastructure
export SWIGGY_DIR=~/swiggy-clone

7.1 Make Persistent Volumes Work

The EBS CSI driver is installed, but Kubernetes still needs a StorageClass that uses it. Create a gp3 class and make it the only default:

Bash

kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF
 
## Two default classes cause confusing behaviour - un-default the old gp2 class
kubectl patch storageclass gp2 \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
 
kubectl get storageclass

7.2 Push Docker Images to ECR

Bash

cd $INFRA_DIR/environments/production
 
## terraform output of a map must use -json (not -raw)
BACKEND_REPO=$(terraform output -json ecr_repository_urls | jq -r '.["swiggy-backend"]')
FRONTEND_REPO=$(terraform output -json ecr_repository_urls | jq -r '.["swiggy-frontend"]')
echo "Backend repo:  ${BACKEND_REPO}"
echo "Frontend repo: ${FRONTEND_REPO}"
 
## Log in to the REGISTRY (the host part), not to one repository.
## The login token is valid for 12 hours.
REGISTRY=${BACKEND_REPO%%/*}
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin ${REGISTRY}
 
cd $SWIGGY_DIR
docker build -t ${BACKEND_REPO}:v1.0.0 ./backend
docker push ${BACKEND_REPO}:v1.0.0
 
docker build -t ${FRONTEND_REPO}:v1.0.0 ./frontend
docker push ${FRONTEND_REPO}:v1.0.0

7.3 Update Manifests and Deploy

Bash

cd $SWIGGY_DIR
 
## Point the manifests at your ECR images
sed -i "s|your-registry/swiggy-backend:v1.0.0|${BACKEND_REPO}:v1.0.0|g" \
  k8s/backend/deployment.yaml
sed -i "s|your-registry/swiggy-frontend:v1.0.0|${FRONTEND_REPO}:v1.0.0|g" \
  k8s/frontend/deployment.yaml
 
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/database/
kubectl apply -f k8s/cache/
kubectl apply -f k8s/backend/
kubectl apply -f k8s/frontend/

Tip
This cluster has no ingress controller yet. If Capstone 1's k8s/ingress/ relies on one (for example ingress-nginx), install that controller first, then apply k8s/ingress/. Otherwise the Ingress object is created but nothing serves it.

Bash

kubectl get pods -n swiggy-clone
kubectl get pvc -n swiggy-clone      ## should be Bound, not Pending

7.4 Create the IRSA Service Account and Prove It Works

Bash

IRSA_ROLE_ARN=$(cd $INFRA_DIR/environments/production && terraform output -raw backend_irsa_role_arn)
 
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend-sa
  namespace: swiggy-clone
  annotations:
    # This annotation links the Service Account to the IAM role
    eks.amazonaws.com/role-arn: ${IRSA_ROLE_ARN}
EOF
 
## Make the backend use it. This triggers a rollout, so new pods get the credentials.
kubectl patch deployment backend -n swiggy-clone \
  -p '{"spec":{"template":{"spec":{"serviceAccountName":"backend-sa"}}}}'
kubectl rollout status deployment/backend -n swiggy-clone

Now prove it, rather than assuming. Run a throwaway pod that uses the same Service Account:

Bash

kubectl run irsa-test -n swiggy-clone --rm -it --restart=Never \
  --image=amazon/aws-cli \
  --overrides='{"spec":{"serviceAccountName":"backend-sa"}}' \
  -- sts get-caller-identity
## "Arn" should contain: assumed-role/platform-production-backend-s3/...
## There are no access keys in the pod - these are temporary credentials.
 
BUCKET=$(cd $INFRA_DIR/environments/production && terraform output -raw assets_bucket)
 
kubectl run irsa-test2 -n swiggy-clone --rm -it --restart=Never \
  --image=amazon/aws-cli \
  --overrides='{"spec":{"serviceAccountName":"backend-sa"}}' \
  -- s3 ls s3://${BUCKET}
## Succeeds (empty listing is fine)
 
kubectl run irsa-test3 -n swiggy-clone --rm -it --restart=Never \
  --image=amazon/aws-cli \
  --overrides='{"spec":{"serviceAccountName":"backend-sa"}}' \
  -- s3 ls
## AccessDenied - the role can use ONE bucket, not list all of them.
## This denial is the proof that least privilege is working.

Tip
Screenshot the three results (redact your account ID). They make a strong portfolio artifact: allowed on its own bucket, denied everywhere else.

Common Mistake
Testing IRSA with pods that started before the Service Account was annotated. Those pods have no credentials until they restart.

Milestone 8: Practice Real Terraform Work in Staging

This milestone rehearses state recovery, import, refactoring and cost-control tasks in a staging environment so mistakes stay cheap.

Why This Section Exists

Most Terraform tutorials end at terraform apply. Real Platform Engineering work starts there. State gets corrupted. Resources get created by hand outside Terraform. Code gets reorganised. Knowing how to handle these without panic is what separates an engineer from someone who followed a tutorial.

Tip
do every drill in staging, never in production. We create staging first because it has its own state file, so a mistake cannot touch production. (Note: ECR repository names are unique per account and region, so staging does not create ECR repositories.)

8.0 Create a Small Staging Environment

HCL

# environments/staging/main.tf
 
terraform {
  required_version = ">= 1.5.0"
 
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }
 
  backend "s3" {
    bucket         = "terraform-state-YOUR_ACCOUNT_ID-ap-south-1"
    key            = "staging/terraform.tfstate"   # different key = separate state
    region         = "ap-south-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}
 
provider "aws" {
  region = "ap-south-1"
 
  default_tags {
    tags = {
      ManagedBy   = "terraform"
      Environment = "staging"
      Project     = "platform-engineering"
    }
  }
}
 
locals {
  cluster_name = "platform-staging"
  environment  = "staging"
}
 
module "vpc" {
  source = "../../modules/vpc"
 
  cluster_name         = local.cluster_name
  environment          = local.environment
  vpc_cidr             = "10.1.0.0/16"
  availability_zones   = ["ap-south-1a", "ap-south-1b"]
  public_subnet_cidrs  = ["10.1.0.0/24", "10.1.1.0/24"]
  private_subnet_cidrs = ["10.1.16.0/20", "10.1.32.0/20"]
 
  # Staging uses one shared NAT gateway to save money
  single_nat_gateway = true
}
 
module "eks" {
  source = "../../modules/eks"
 
  cluster_name       = local.cluster_name
  environment        = local.environment
  private_subnet_ids = module.vpc.private_subnet_ids
  public_subnet_ids  = module.vpc.public_subnet_ids
 
  node_desired_size = 1
  node_min_size     = 1
  node_max_size     = 2
}

Bash

cd $INFRA_DIR/environments/staging
terraform init
terraform plan -out=staging.tfplan
terraform apply staging.tfplan

8.1 Understanding and Inspecting State

The state file maps every resource in your .tf files to a real AWS resource ID. Without it, Terraform does not know what it created.

Bash

## List everything Terraform tracks
terraform state list
## module.vpc.aws_vpc.main
## module.vpc.aws_subnet.public[0]
## module.eks.aws_eks_cluster.main
## ...
 
## Inspect one resource in detail
terraform state show module.eks.aws_eks_cluster.main

8.2 Breaking and Recovering State

Drill A: the state file was "deleted". Because the bucket is versioned, a delete only adds a delete marker. Recovery is to remove that marker.

Bash

export TF_STATE_BUCKET="terraform-state-${ACCOUNT_ID}-ap-south-1"
 
## Simulate the accident (staging only!)
aws s3 rm s3://${TF_STATE_BUCKET}/staging/terraform.tfstate
 
terraform plan
## It now behaves as if nothing exists and wants to create everything again.
## DO NOT APPLY - that would try to create duplicates.
 
## Find the delete marker
aws s3api list-object-versions \
  --bucket ${TF_STATE_BUCKET} \
  --prefix staging/terraform.tfstate \
  --query 'DeleteMarkers[?IsLatest].{Key:Key,VersionId:VersionId}'
 
## Remove the marker - the previous version becomes current again
aws s3api delete-object \
  --bucket ${TF_STATE_BUCKET} \
  --key staging/terraform.tfstate \
  --version-id MARKER_VERSION_ID
 
terraform plan
## "No changes" - recovered

Tip
If the state was overwritten instead of deleted, use aws s3api list-object-versions to find an older good version and copy it back with aws s3api copy-object --copy-source "BUCKET/KEY?versionId=VERSION_ID". This is the reason versioning was turned on in Milestone 1.

Drill B: a stuck lock. If a run is killed halfway, the lock can stay behind and every later run fails with "Error acquiring the state lock" and a lock ID:

Bash

## Only after you are SURE nobody else is running Terraform:
terraform force-unlock LOCK_ID_FROM_THE_ERROR

Drill C: drift. Someone changed a resource in the console. Compare reality to state without changing anything:

Bash

## Shows differences between real AWS and the state file, changes nothing
terraform plan -refresh-only
 
## If you accept the real-world values into state:
terraform apply -refresh-only

Tip
You may see older material using terraform refresh. It is deprecated; -refresh-only is the safer replacement because it shows you the changes before recording them. Whether the right fix is "update the code" or "undo the console change" is a team decision, not a Terraform one.

8.3 Importing Existing Resources

Scenario: someone created an S3 bucket by hand. You want Terraform to manage it from now on. Importing says: "This real resource already exists. Start tracking it."

Bash

## Create the "mystery" bucket manually, as a colleague would have
aws s3api create-bucket \
  --bucket swiggy-clone-import-demo-${ACCOUNT_ID} \
  --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1

Write the resource and an import block (Terraform 1.5+) in environments/staging/imports.tf:

HCL

# environments/staging/imports.tf
 
resource "aws_s3_bucket" "import_demo" {
  bucket = "swiggy-clone-import-demo-YOUR_ACCOUNT_ID"
}
 
import {
  to = aws_s3_bucket.import_demo
  id = "swiggy-clone-import-demo-YOUR_ACCOUNT_ID"
}

Bash

terraform plan
## Shows the bucket being imported. Any extra "~" changes mean your code differs
## from the real bucket - edit the code until the plan is clean.
 
terraform apply
## The bucket is now tracked in state
terraform state list | grep import_demo

Once it is imported, you can delete the import block.

8.4 Debugging Dependency Problems

Cycle error. If two resources reference each other, Terraform stops with Error: Cycle: ... and lists the resources involved. There is nothing to "force": you remove the circular reference. Typically one side only needs an ID, not the whole other resource, or a security-group rule should be split into its own resource.

Order problems. Terraform normally infers order from references. When there is a hidden dependency (something that must exist first but is not referenced), declare it explicitly. Inside a module you can only depend on the module as a whole:

HCL

module "eks" {
  source = "../../modules/eks"
  # ...
  depends_on = [module.vpc]
}

Targeted applies. -target limits a run to selected resources. Treat it as an emergency tool, not a habit, because it skips the full dependency check:

Bash

terraform apply -target=module.vpc
terraform apply            ## always finish with a normal, full apply

To see the dependency graph, terraform graph prints DOT format. Pipe it to Graphviz (dot -Tpng) if you have it installed.

8.5 Moving and Removing Resources in State

Scenario: you reorganised your code. A resource moved into a module. Without help, Terraform plans to destroy the old one and create a new one. You do not want to destroy a cluster just because you renamed something.

The modern, reviewable way is a moved block in code:

HCL

moved {
  from = aws_eks_cluster.main
  to   = module.eks.aws_eks_cluster.main
}

terraform plan then shows it as a move, not a destroy-and-create. The older, one-off command does the same by hand:

Bash

terraform state mv aws_eks_cluster.main module.eks.aws_eks_cluster.main

Prefer moved blocks because they live in Git and your teammates' runs pick them up too.

Scenario: forget a resource without deleting it in AWS. Quote addresses that contain brackets and quotes, or your shell will mangle them:

Bash

## Stops Terraform tracking it. Does NOT delete the real resource.
terraform state rm 'module.ecr.aws_ecr_repository.main["swiggy-backend"]'
## A later plan will want to create it again, unless you import it back.

8.6 Targeted Destroys for Cost Control

A company might shut staging nodes down overnight to save money.

Bash

cd $INFRA_DIR/environments/staging
 
## Destroy only the node group (stops the EC2 cost; control plane and VPC remain)
terraform destroy -target=module.eks.aws_eks_node_group.main
 
## Recreate it in the morning
terraform apply -target=module.eks.aws_eks_node_group.main

Tip
Scaling the node group's desired_size to 0 is often a cleaner way to pause a cluster than destroying the node group. Either way, remember the control plane and NAT gateway keep billing while the cluster exists.

Common Mistake
Running these drills against production state. A wrong state rm or force-unlock there can cause a real outage.

Validation & Testing

This step verifies the infrastructure against a production checklist, then covers cleanup, common mistakes and debugging.

Production Checklist

Bash

cd $INFRA_DIR/environments/production
 
## 1. VPC has public and private subnets
aws ec2 describe-subnets \
  --filters "Name=tag:Environment,Values=production" \
  --query 'Subnets[*].{ID:SubnetId,AZ:AvailabilityZone,Public:MapPublicIpOnLaunch,CIDR:CidrBlock}' \
  --output table
## Expect public subnets (Public=True) and private subnets (Public=False)
 
## 2. EKS nodes are in private subnets (no public IPs)
kubectl get nodes -o wide
## EXTERNAL-IP should be <none> for every node
 
## 3. Remote state is configured
jq '.backend.type' .terraform/terraform.tfstate
## "s3"
 
## 4. Running images come from ECR
kubectl get deployment backend -n swiggy-clone \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
## Should contain .dkr.ecr.
 
## 5. No static AWS keys in any pod
kubectl get pods -n swiggy-clone -o json | \
  jq '.items[].spec.containers[].env[]? | select(.name | test("AWS_(SECRET|ACCESS)"))'
## Should print nothing
 
## 6. IRSA variables were injected into the backend pod
kubectl exec -n swiggy-clone deploy/backend -- env | grep AWS_
## Expect AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE (and no access key variables)
 
## 7. State bucket is encrypted and private
aws s3api get-bucket-encryption --bucket terraform-state-${ACCOUNT_ID}-ap-south-1
aws s3api get-public-access-block --bucket terraform-state-${ACCOUNT_ID}-ap-south-1
 
## 8. Review how exposed the Kubernetes API is
aws eks describe-cluster --name platform-production \
  --query 'cluster.resourcesVpcConfig.{Public:endpointPublicAccess,Private:endpointPrivateAccess,Cidrs:publicAccessCidrs}'
## Decide: should this still be 0.0.0.0/0 for a real production cluster?

Cleanup: Do This the Same Day

Resources created by Kubernetes (load balancers, EBS volumes) are not in Terraform's state. If you destroy the VPC first, those leftovers block the destroy. Remove them first:

Bash

## 1. Delete everything Kubernetes created, so its load balancers and volumes go away
kubectl delete namespace swiggy-clone
## Wait until the namespace is gone and any load balancers are deleted.
## Check: aws elbv2 describe-load-balancers --region ap-south-1
 
## 2. Destroy staging, then production
cd $INFRA_DIR/environments/staging    && terraform destroy
cd $INFRA_DIR/environments/production && terraform destroy

terraform destroy on production will fail on ECR repositories that still contain images, and on the non-empty S3 bucket. Empty them first (aws ecr batch-delete-image, aws s3 rm --recursive), or add force_delete = true to the ECR repository and force_destroy = true to the bucket for learning environments.

Finally, delete the state bucket (empty all versions first) and the DynamoDB table terraform-state-lock. Then check the AWS Billing console the next day to confirm nothing is still running.

Common Mistakes

Common Mistake
Storing Terraform state locally. Two engineers run terraform apply at the same time. Both read the same state, both plan, both apply. One creates resources the other does not know about. State is corrupted, and neither can run Terraform safely until someone repairs it by hand. Fix: Use remote state with locking from day one. Ten minutes of setup saves hours of pain.

Common Mistake
Putting EKS nodes in public subnets. For simplicity a developer puts nodes in public subnets. They get public IPs. A scan finds the kubelet port reachable from the internet. If it is exploited, an attacker can reach pods, secrets and service-account tokens on that node. Fix: Nodes always go in private subnets. Only load balancers go in public subnets.

Common Mistake
Long-lived AWS keys in Kubernetes Secrets. A developer needs a pod to write to S3, so they create an IAM user and store the keys in a Secret. Months later the keys have broad permissions, have never been rotated, and sit in several places. A compromised pod means a compromised AWS account. Fix: Use IRSA, with a role scoped to exactly what the workload calls, as you proved in Milestone 7.

Common Mistake
Running terraform apply without reading the plan. A "small" change to the VPC module forces replacement of other resources. The plan said -/+ next to them. Nobody read it. Pods are evicted and the service is down. Fix: Read every line. Treat - and -/+ as stop signs until you understand why. Apply the saved plan file you reviewed.

Common Mistake
One huge Terraform configuration for everything. Network, cluster, databases and twenty services in one state. Every plan takes ages, and a small change to a bucket can accidentally touch the network. The blast radius of any mistake is everything. Fix: Split into layers (network, cluster, per-service) with their own state, so a change in one layer cannot reach the others.

Common Mistake
Skipping the EBS CSI driver on EKS. The database pod stays Pending because its PersistentVolumeClaim is never provisioned. Nothing in the pod logs explains why. Fix: Install the EBS CSI add-on with its IRSA role (Milestone 3), create a StorageClass that uses it (Milestone 7), and check kubectl get pvc.

Common Mistake
Destroying the VPC while Kubernetes load balancers still exist. terraform destroy hangs or fails on subnets and the VPC because load balancers and network interfaces created by Kubernetes still exist outside Terraform's state. Fix: Delete the Kubernetes workloads first, wait for the load balancers to disappear, then destroy.

Debugging Playbook

terraform plan shows an unexpected destroy:

Bash

## Find out WHY - Terraform always names the attribute that forces replacement
terraform plan 2>&1 | grep -B2 -A6 "forces replacement"
 
## Immutable fields (for example a subnet's availability zone or a VPC's CIDR block)
## cannot change in place. Revert the code change or plan a migration.
 
## Make sure state matches reality
terraform plan -refresh-only
terraform state list
terraform state show module.eks.aws_eks_cluster.main

EKS nodes not joining the cluster:

Bash

aws eks describe-nodegroup \
  --cluster-name platform-production \
  --nodegroup-name platform-production-nodes \
  --query 'nodegroup.{Status:status,Issues:health.issues}'
 
## Common causes:
## - Node role missing one of the three required policies
## - Private subnets have no route to a NAT gateway (nodes cannot reach the EKS API)
## - Subnets in the wrong VPC or missing required tags
## - Instance capacity or quota problems in the region

A pod cannot call AWS (IRSA not working):

Bash

## 1. Does the Service Account carry the annotation?
kubectl get serviceaccount backend-sa -n swiggy-clone -o yaml | grep role-arn
 
## 2. Is the pod using that Service Account? (pods started BEFORE the change must be restarted)
kubectl get pod <pod-name> -n swiggy-clone -o jsonpath='{.spec.serviceAccountName}'
 
## 3. Were the IRSA variables injected?
kubectl exec -n swiggy-clone <pod-name> -- env | grep AWS_
## Expect AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE
 
## 4. Does the OIDC provider exist for THIS cluster?
aws iam list-open-id-connect-providers
 
## Common causes of "Not authorized to perform sts:AssumeRoleWithWebIdentity":
## - Trust policy still has "https://" in the condition key (it must be removed)
## - Wrong namespace or Service Account name in the trust policy
## - Wrong role ARN in the annotation
## - The pod was not restarted after the Service Account changed

PersistentVolumeClaim stuck in Pending:

Bash

kubectl describe pvc <name> -n swiggy-clone
kubectl get pods -n kube-system | grep ebs-csi
kubectl get storageclass
## Check: add-on running? StorageClass uses ebs.csi.aws.com? Only one default class?

What You Have Built

You provisioned real cloud infrastructure the way professional Platform Engineers do.

✅ A VPC with public and private subnets across 2 availability zones
✅ NAT gateway(s) giving private nodes outbound internet access
✅ An EKS cluster whose nodes run in private subnets
✅ An OIDC provider and two IRSA roles (EBS CSI driver, backend pods)
✅ Backend pods that can use one S3 bucket and nothing else - proven, not assumed
✅ ECR repositories with image scanning and lifecycle rules
✅ Remote state in S3 with locking
✅ Reusable modules that built both a production and a staging environment
✅ Practice recovering from broken state, importing resources and moving code
✅ Capstone 1's application running on infrastructure you provisioned

More important than the checklist: you know why each decision was made. Why nodes live in private subnets, why IRSA replaces stored keys, why state is remote and locked, why modules are separate from environments, and why you rehearse failures in staging.

Next: Capstone 3 - Build a GitOps Platform with ArgoCD. You will deploy what you built in Capstones 1 and 2 through GitOps delivery, with multi-environment promotion, App of Apps, and progressive delivery.

Found a mistake, an outdated step or a broken link?

Build Production AWS Infrastructure with Terraform

Blueprint Walkthrough

Architecture Overview & Problem Statement

What You Build

What Is Terraform and Why Not Just Use the AWS Console?

Milestone 1: Set Up the Project and Remote State

Why Remote State?

1.1 Project Structure

1.2 Create the Remote State Infrastructure

Milestone 2: Build the VPC Module

What Is a VPC and Why Do We Need One?

2.1 VPC Module

Milestone 3: Build the EKS Module

What Is EKS and What Does AWS Manage for You?

What Is IRSA and Why Does It Matter?

3.1 IAM Roles for EKS

3.2 EKS Cluster, Node Group and EBS CSI Driver

Milestone 4: Build the ECR Module

What Is ECR and Why Not Use Docker Hub?

Milestone 5: Configure the Production Environment

Connecting Everything Together

Milestone 6: Deploy the Infrastructure

The Terraform Workflow

Connect kubectl to Your New Cluster

Milestone 7: Push Images and Deploy Capstone 1

7.1 Make Persistent Volumes Work

7.2 Push Docker Images to ECR

7.3 Update Manifests and Deploy

7.4 Create the IRSA Service Account and Prove It Works

Milestone 8: Practice Real Terraform Work in Staging

Why This Section Exists

8.0 Create a Small Staging Environment

8.1 Understanding and Inspecting State

8.2 Breaking and Recovering State

8.3 Importing Existing Resources

8.4 Debugging Dependency Problems

8.5 Moving and Removing Resources in State

8.6 Targeted Destroys for Cost Control

Validation & Testing

Production Checklist

Cleanup: Do This the Same Day

Common Mistakes

Debugging Playbook

What You Have Built