Multi-Region Active-Active on AWS with Terraform

Deploy the same app in Mumbai and Singapore with Terraform, Route 53 latency routing and Aurora Global Database, then measure your failover time.

Architecture Overview & Problem Statement

This project runs the same application stack in two AWS regions, Mumbai (ap-south-1) and Singapore (ap-southeast-1), and sends each user to the nearer one with Route 53 latency routing. A Route 53 health check removes a failing region from DNS, and an Aurora Global Database keeps a second copy of the data in Singapore that you can promote. One Terraform configuration builds everything. You finish with a failover drill where you inject a regional failure and measure how long recovery really took, which is the evidence to show in a portfolio.

You need Terraform, an AWS account, and a domain whose DNS is hosted in Route 53 (a delegated subdomain is enough). Do the single-region projects first, such as AWS 3-Tier on EC2 with Terraform and CI/CD, because this project reuses those ideas in two regions at once.

What you build:

A reusable regional module (VPC, ALB, Auto Scaling group) called once per region
An Aurora Global Database with a writer in Mumbai and a secondary cluster in Singapore
Route 53 latency records with health checks
A timed failover drill for both traffic and the database

flowchart TD U[Users] --> R53[Route 53
latency routing] HC[Health checks] -.->|probe /health| R53 R53 --> M[Mumbai stack
ALB and Auto Scaling] R53 --> S[Singapore stack
ALB and Auto Scaling] M --> DB1[(Aurora writer
Mumbai)] S --> DB2[(Aurora secondary
Singapore, read only)] DB1 -->|replicates| DB2 classDef dns fill:#8B5CF6,color:#fff classDef app fill:#326CE5,color:#fff classDef data fill:#10B981,color:#fff classDef probe fill:#475569,color:#fff class R53 dns class M,S app class DB1,DB2 data class HC,U probe

Be precise about "active-active"

The application tier here is truly active-active: both regions serve live traffic at the same time. The database tier is not. Aurora Global Database has one writer region, and the Singapore cluster is read-only until you promote it. So writes from Singapore users either travel to Mumbai or need extra design, and a regional failure means promoting the secondary, which is a separate step with its own timing.

This matters in an interview. Call it what it is: active-active compute with a single-writer database and a promotable replica.

What a failover actually takes

A regional failure has several parts, and each adds time:

Stage	What happens	Rough size
Detection	Health checks fail 3 times, 10 seconds apart	about 30 seconds
DNS change	Route 53 stops returning the failed region	a few seconds
Record TTL	Resolvers may keep the old answer for up to the TTL	up to 30 seconds here
Client caching	Some clients and resolvers ignore low TTLs	varies, can be minutes
Database promotion	Singapore is promoted to writer	measured, often a minute or more

Added together, a DNS-level failover taking about a minute or two is realistic, and "under 60 seconds" is not something the design guarantees. Treat any such figure as a target and measure it. Milestone 6 has you do that, and record the real number.

Security
Cross-region replication and a second full stack double your cost. Aurora Global needs large instance classes, and each region has a NAT Gateway and a load balancer. Destroy the stack when you finish a session.

Milestone 1: Create the Project and Choose Terraform's Structure

This milestone sets up one root configuration with two provider aliases and a shared module. It does not use Terraform workspaces for the regions. Workspaces give the same code separate state, but Route 53 and the Aurora Global Database depend on resources in both regions, and that is much easier to express when both regions live in one configuration and one state.

Bash

mkdir multi-region && cd multi-region
mkdir -p modules/region

Create main.tf:

HCL

terraform {
  required_version = ">= 1.11.0"
 
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.79"
    }
  }
 
  backend "s3" {
    bucket       = "YOUR_STATE_BUCKET"
    key          = "multi-region/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}
 
provider "aws" {
  alias  = "mumbai"
  region = "ap-south-1"
}
 
provider "aws" {
  alias  = "singapore"
  region = "ap-southeast-1"
}
 
module "mumbai" {
  source      = "./modules/region"
  providers   = { aws = aws.mumbai }
  name        = "mumbai"
  region_name = "ap-south-1"
  vpc_cidr    = "10.0.0.0/16"
  azs         = ["ap-south-1a", "ap-south-1b"]
}
 
module "singapore" {
  source      = "./modules/region"
  providers   = { aws = aws.singapore }
  name        = "singapore"
  region_name = "ap-southeast-1"
  vpc_cidr    = "10.1.0.0/16"
  azs         = ["ap-southeast-1a", "ap-southeast-1b"]
}

The two VPC ranges do not overlap, which keeps the option of connecting the regions later.

Milestone 2: Build the Regional Module

This milestone defines one complete regional stack. It runs twice, once per region, from the same code. Each instance runs a tiny web server with two endpoints: /health for the load balancer and Route 53, and /region, which returns the region name so you can see where a request landed.

Create modules/region/main.tf:

HCL

variable "name" {
  type = string
}
 
variable "region_name" {
  type = string
}
 
variable "vpc_cidr" {
  type = string
}
 
variable "azs" {
  type = list(string)
}
 
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
 
  name = var.name
  cidr = var.vpc_cidr
  azs  = var.azs
 
  public_subnets   = [for i, az in var.azs : cidrsubnet(var.vpc_cidr, 8, i)]
  private_subnets  = [for i, az in var.azs : cidrsubnet(var.vpc_cidr, 8, i + 10)]
  database_subnets = [for i, az in var.azs : cidrsubnet(var.vpc_cidr, 8, i + 20)]
 
  create_database_subnet_group = true
  enable_nat_gateway           = true
  single_nat_gateway           = true
  enable_dns_hostnames         = true
}
 
resource "aws_security_group" "alb" {
  name_prefix = "${var.name}-alb-"
  vpc_id      = module.vpc.vpc_id
 
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
 
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
 
resource "aws_security_group" "app" {
  name_prefix = "${var.name}-app-"
  vpc_id      = module.vpc.vpc_id
 
  ingress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }
 
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
 
resource "aws_security_group" "db" {
  name_prefix = "${var.name}-db-"
  vpc_id      = module.vpc.vpc_id
 
  ingress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id]
  }
}
 
data "aws_ssm_parameter" "al2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
}
 
resource "aws_launch_template" "app" {
  name_prefix            = "${var.name}-app-"
  image_id               = data.aws_ssm_parameter.al2023.value
  instance_type          = "t3.micro"
  vpc_security_group_ids = [aws_security_group.app.id]
 
  user_data = base64encode(<<-EOF
    #!/bin/bash
    dnf install -y nginx
    echo "${var.region_name}" > /usr/share/nginx/html/region
    echo "ok" > /usr/share/nginx/html/health
    systemctl enable --now nginx
  EOF
  )
 
  metadata_options {
    http_tokens = "required"
  }
}
 
resource "aws_lb" "main" {
  name               = "${var.name}-alb"
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = module.vpc.public_subnets
}
 
resource "aws_lb_target_group" "app" {
  name     = "${var.name}-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = module.vpc.vpc_id
 
  health_check {
    path                = "/health"
    interval            = 10
    timeout             = 5
    healthy_threshold   = 2
    unhealthy_threshold = 2
  }
}
 
resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.main.arn
  port              = 80
  protocol          = "HTTP"
 
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}
 
resource "aws_autoscaling_group" "app" {
  name                = "${var.name}-app"
  vpc_zone_identifier = module.vpc.private_subnets
  target_group_arns   = [aws_lb_target_group.app.arn]
  health_check_type   = "ELB"
  min_size            = 2
  max_size            = 6
  desired_capacity    = 2
 
  launch_template {
    id      = aws_launch_template.app.id
    version = aws_launch_template.app.latest_version
  }
}
 
output "alb_dns_name" {
  value = aws_lb.main.dns_name
}
 
output "listener_arn" {
  value = aws_lb_listener.http.arn
}
 
output "target_group_arn" {
  value = aws_lb_target_group.app.arn
}
 
output "db_subnet_group_name" {
  value = module.vpc.database_subnet_group_name
}
 
output "db_security_group_id" {
  value = aws_security_group.db.id
}

Create outputs.tf in the root so you can reach each region's values later:

HCL

output "mumbai_alb_dns_name" {
  value = module.mumbai.alb_dns_name
}
 
output "mumbai_listener_arn" {
  value = module.mumbai.listener_arn
}
 
output "mumbai_target_group_arn" {
  value = module.mumbai.target_group_arn
}
 
output "singapore_alb_dns_name" {
  value = module.singapore.alb_dns_name
}
 
output "singapore_listener_arn" {
  value = module.singapore.listener_arn
}
 
output "singapore_target_group_arn" {
  value = module.singapore.target_group_arn
}

Bash

terraform init
terraform apply -target=module.mumbai -target=module.singapore
 
curl http://$(terraform output -raw mumbai_alb_dns_name)/region
curl http://$(terraform output -raw singapore_alb_dns_name)/region
# Expected: ap-south-1, then ap-southeast-1

Wait a few minutes after the apply, because the instances need time to install the web server and pass health checks.

Common Mistake
Reusing the same VPC CIDR in both regions. It works until you ever need to connect the regions, and then it cannot be fixed without rebuilding the network. Pick non-overlapping ranges from the start.

Milestone 3: Add the Aurora Global Database

This milestone creates the data layer. An Aurora global cluster links a primary cluster in one region to read-only secondary clusters in others, replicating storage across regions.

Pick an Aurora PostgreSQL version that supports Global Database in both regions instead of guessing one:

Bash

aws rds describe-db-engine-versions \
  --engine aurora-postgresql \
  --region ap-south-1 \
  --query 'DBEngineVersions[?SupportsGlobalDatabases==`true`].EngineVersion' \
  --output text

Create aurora.tf and set aurora_version to a version from that list, available in both regions:

HCL

variable "aurora_version" {
  type = string
}
 
resource "aws_rds_global_cluster" "main" {
  provider                  = aws.mumbai
  global_cluster_identifier = "app-global"
  engine                    = "aurora-postgresql"
  engine_version            = var.aurora_version
  storage_encrypted         = true
}
 
resource "aws_rds_cluster" "mumbai" {
  provider                    = aws.mumbai
  cluster_identifier          = "app-mumbai"
  engine                      = aws_rds_global_cluster.main.engine
  engine_version              = aws_rds_global_cluster.main.engine_version
  global_cluster_identifier   = aws_rds_global_cluster.main.id
  database_name               = "appdb"
  master_username             = "dbadmin"
  manage_master_user_password = true
  storage_encrypted           = true
  db_subnet_group_name        = module.mumbai.db_subnet_group_name
  vpc_security_group_ids      = [module.mumbai.db_security_group_id]
  skip_final_snapshot         = true
}
 
resource "aws_rds_cluster_instance" "mumbai" {
  provider           = aws.mumbai
  identifier         = "app-mumbai-1"
  cluster_identifier = aws_rds_cluster.mumbai.id
  instance_class     = "db.r6g.large"
  engine             = aws_rds_cluster.mumbai.engine
  engine_version     = aws_rds_cluster.mumbai.engine_version
}
 
# The secondary cluster needs an encryption key in its own region
data "aws_kms_key" "rds_singapore" {
  provider = aws.singapore
  key_id   = "alias/aws/rds"
}
 
resource "aws_rds_cluster" "singapore" {
  provider                  = aws.singapore
  cluster_identifier        = "app-singapore"
  engine                    = aws_rds_global_cluster.main.engine
  engine_version            = aws_rds_global_cluster.main.engine_version
  global_cluster_identifier = aws_rds_global_cluster.main.id
  storage_encrypted         = true
  kms_key_id                = data.aws_kms_key.rds_singapore.arn
  db_subnet_group_name      = module.singapore.db_subnet_group_name
  vpc_security_group_ids    = [module.singapore.db_security_group_id]
  skip_final_snapshot       = true
 
  # The primary must have an instance before a secondary can attach
  depends_on = [aws_rds_cluster_instance.mumbai]
 
  lifecycle {
    ignore_changes = [replication_source_identifier]
  }
}
 
resource "aws_rds_cluster_instance" "singapore" {
  provider           = aws.singapore
  identifier         = "app-singapore-1"
  cluster_identifier = aws_rds_cluster.singapore.id
  instance_class     = "db.r6g.large"
  engine             = aws_rds_cluster.singapore.engine
  engine_version     = aws_rds_cluster.singapore.engine_version
}

The secondary cluster has no master username or password, because it receives them from the primary. AWS generates and stores the primary's password in Secrets Manager, so no database password appears in your code or state.

Bash

terraform apply
aws rds describe-global-clusters --global-cluster-identifier app-global \
  --query 'GlobalClusters[0].GlobalClusterMembers[*].[DBClusterArn,IsWriter]' --output table
# Expected: the Mumbai cluster is the writer, Singapore is not

Expect this apply to take 20 minutes or more.

Common Mistake
Trying to create the secondary cluster before the primary has a running instance. The attach fails with a confusing error. The depends_on above exists for exactly this reason.

Milestone 4: Add Route 53 Latency Routing and Health Checks

This milestone connects users to the stack. A latency record returns the answer for the region with the lowest network latency to the user. A health check probes each ALB, and Route 53 stops returning a record whose check fails.

Create route53.tf. The records are CNAMEs with a 30-second TTL: a low TTL shortens how long resolvers keep a stale answer, at the price of more DNS queries.

HCL

variable "domain_name" {
  type        = string
  description = "For example api.example.com, inside a hosted zone you control"
}
 
variable "hosted_zone_id" {
  type = string
}
 
resource "aws_route53_health_check" "mumbai" {
  provider          = aws.mumbai
  fqdn              = module.mumbai.alb_dns_name
  port              = 80
  type              = "HTTP"
  resource_path     = "/health"
  request_interval  = 10
  failure_threshold = 3
  tags              = { Name = "mumbai" }
}
 
resource "aws_route53_health_check" "singapore" {
  provider          = aws.singapore
  fqdn              = module.singapore.alb_dns_name
  port              = 80
  type              = "HTTP"
  resource_path     = "/health"
  request_interval  = 10
  failure_threshold = 3
  tags              = { Name = "singapore" }
}
 
resource "aws_route53_record" "mumbai" {
  provider        = aws.mumbai
  zone_id         = var.hosted_zone_id
  name            = var.domain_name
  type            = "CNAME"
  ttl             = 30
  set_identifier  = "mumbai"
  health_check_id = aws_route53_health_check.mumbai.id
  records         = [module.mumbai.alb_dns_name]
 
  latency_routing_policy {
    region = "ap-south-1"
  }
}
 
resource "aws_route53_record" "singapore" {
  provider        = aws.singapore
  zone_id         = var.hosted_zone_id
  name            = var.domain_name
  type            = "CNAME"
  ttl             = 30
  set_identifier  = "singapore"
  health_check_id = aws_route53_health_check.singapore.id
  records         = [module.singapore.alb_dns_name]
 
  latency_routing_policy {
    region = "ap-southeast-1"
  }
}

Bash

terraform apply
dig +short $DOMAIN_NAME
curl http://$DOMAIN_NAME/region

The answer depends on where you are. Note which region your own machine reaches, because that is the region you will fail in Milestone 6. Fast health checks (10-second interval) cost a little extra per check, which is another cost-versus-speed trade-off.

Tip
Alias records to a load balancer have a fixed TTL that you cannot lower, so they can be slower to fail over than the CNAME records above. This project uses CNAMEs so the timing is under your control.

Milestone 5: Write the Failover Runbook Before You Need It

This milestone documents what to do when a region fails, written now and not during an incident. Put this in your repository as RUNBOOK.md.

Traffic. Route 53 handles it automatically. Your job is to confirm: check the health status and that users reach the surviving region.

Bash

aws route53 get-health-check-status --health-check-id MUMBAI_HEALTH_CHECK_ID
curl http://$DOMAIN_NAME/region

Database. Promotion is not automatic here. For a real regional outage, the primary cluster cannot take part in an orderly handover, so you promote the secondary and accept that it may be missing the most recent writes:

Bash

aws rds failover-global-cluster \
  --region ap-southeast-1 \
  --global-cluster-identifier app-global \
  --target-db-cluster-identifier arn:aws:rds:ap-southeast-1:ACCOUNT_ID:cluster:app-singapore \
  --allow-data-loss

Run the call against the surviving region's endpoint, since the failed region may not answer. A planned switchover, when both regions are healthy, uses a different mode with no data loss, so use that for maintenance and drills that do not simulate a real outage. Check the current AWS CLI reference for the exact option names in your version.

Then point the application's database connection at the new writer. The demo app here is stateless, so your runbook should state how a real application would learn the new writer endpoint, for example through a DNS name that you update, or by reading it from configuration.

Data loss. The amount of data at risk is bounded by replication lag at the moment of failure. Check it regularly:

Bash

aws cloudwatch get-metric-statistics \
  --region ap-southeast-1 \
  --namespace AWS/RDS \
  --metric-name AuroraGlobalDBReplicationLag \
  --dimensions Name=DBClusterIdentifier,Value=app-singapore \
  --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 --statistics Average Maximum

The value is in milliseconds. AWS describes typical lag as low, but record your own average and maximum.

Milestone 6: Run a Timed Failover Drill

This milestone measures what you could only estimate. You will fail the region that your machine currently reaches, and record the timeline.

First, start a probe that requests the app every second and logs a timestamp and the region. Save it as probe.sh:

Bash

#!/bin/bash
# Usage: bash probe.sh api.example.com
while true; do
  TS=$(date +%s.%N)
  BODY=$(curl -sf -m 2 "http://$1/region" || echo "FAIL")
  echo "$TS $BODY"
  sleep 1
done

In one terminal, run bash probe.sh $DOMAIN_NAME | tee probe.log. Note the region it prints, then, in another terminal, inject a failure into that region by making its load balancer answer 503. This example fails Mumbai. Record the time of the injection:

Bash

date +%s.%N   # T0, write this down
 
aws elbv2 modify-listener \
  --region ap-south-1 \
  --listener-arn $(terraform output -raw mumbai_listener_arn) \
  --default-actions 'Type=fixed-response,FixedResponseConfig={StatusCode=503,ContentType=text/plain,MessageBody=down}'

Watch the probe until the other region answers consistently, then restore the listener:

Bash

aws elbv2 modify-listener \
  --region ap-south-1 \
  --listener-arn $(terraform output -raw mumbai_listener_arn) \
  --default-actions "Type=forward,TargetGroupArn=$(terraform output -raw mumbai_target_group_arn)"

Work out the numbers from the log:

Bash

BEFORE=$(head -1 probe.log | awk '{print $2}')
FIRST_FAIL=$(grep -m1 FAIL probe.log | awk '{print $1}')
RECOVERED=$(awk -v r="$BEFORE" '$2!="FAIL" && $2!=r {print $1; exit}' probe.log)
echo "Requests failed for $(echo "$RECOVERED - $FIRST_FAIL" | bc) seconds"

Also check when Route 53 marked the endpoint unhealthy by repeating aws route53 get-health-check-status during the test. Then run the database promotion from Milestone 5, in a separate drill, and time how long until the Singapore cluster reports IsWriter as true:

Bash

aws rds describe-global-clusters --global-cluster-identifier app-global \
  --query 'GlobalClusters[0].GlobalClusterMembers[*].[DBClusterArn,IsWriter]' --output table

Fill in a results table in your README:

Measure	Target	Measured
Failure injected to health check unhealthy	About 30 seconds
Time requests failed for a client	Under 90 seconds
Database promotion to writer	Recorded
Replication lag before the drill (average and maximum)	Under a few seconds

A measured result that misses the target is still a good result, as long as you explain why: the TTL, the health check interval, or a client that cached the old answer. Then say what you would change, such as a lower failure threshold, and what that would cost.

Validation & Testing

Confirm each part of the system before you call it finished.

Bash

# 1. Both regions answer directly and name themselves
curl http://$(terraform output -raw mumbai_alb_dns_name)/region
curl http://$(terraform output -raw singapore_alb_dns_name)/region
 
# 2. DNS returns one of them for the domain
dig +short $DOMAIN_NAME
 
# 3. The global database has one writer and one secondary
aws rds describe-global-clusters --global-cluster-identifier app-global \
  --query 'GlobalClusters[0].GlobalClusterMembers[*].[DBClusterArn,IsWriter]' --output table
 
# 4. Both Route 53 health checks are healthy
aws route53 get-health-check-status --health-check-id MUMBAI_HEALTH_CHECK_ID
 
# 5. Terraform shows no drift
terraform plan

In your portfolio, include the architecture diagram, the completed results table with your own measured numbers, the probe log excerpt around the failure, and the RUNBOOK.md. State plainly which parts are active-active, which are not, and what remaining risks you would address next.

Destroy everything when you finish, since two full stacks and two Aurora instances cost real money:

Bash

terraform destroy

If the destroy complains that the global cluster still has members, run it again after the secondary cluster has been removed. Check the AWS console afterwards for leftover load balancers, NAT Gateways, and database snapshots.

Found a mistake, an outdated step or a broken link?

Multi-Region Active-Active on AWS with Terraform

Blueprint Walkthrough

Architecture Overview & Problem Statement

Be precise about "active-active"

What a failover actually takes

Milestone 1: Create the Project and Choose Terraform's Structure

Milestone 2: Build the Regional Module

Milestone 3: Add the Aurora Global Database

Milestone 4: Add Route 53 Latency Routing and Health Checks

Milestone 5: Write the Failover Runbook Before You Need It

Milestone 6: Run a Timed Failover Drill

Validation & Testing