Deploy Kubernetes Cluster Autoscaler with Spot Instance Cost Optimisation

Configure EKS Cluster Autoscaler with mixed On-Demand and Spot node groups, Node Termination Handler, and pod disruption budgets for 70% cost reduction.

Related Concepts & TermsSubnet Deployment grep IAM Role Taint Auto Scaling Group

Domains & Technologies

Domains

COST-OPTIMISATION

Technologies

KUBERNETESAWSTERRAFORM

Blueprint Walkthrough

Architecture Overview

This project solves one of the most common complaints from DevOps engineers — Kubernetes costs too much. The solution is Spot instances combined with the Cluster Autoscaler. Spot instances are unused AWS capacity sold at up to 90% discount. The catch is AWS can reclaim them with a 2-minute warning. The Node Termination Handler catches that warning and gracefully drains the node before AWS takes it.

This is the cost architecture used by Swiggy and Meesho — stateless application workloads run on Spot (cheap), stateful workloads and system components run on On-Demand (reliable).

◈ DIAGRAM

    EKS Control Plane (managed by AWS)
           |
    +------+--------+
    |               |
On-Demand        Spot Instance
Node Group       Node Group
(t3.medium)      (m5.large, m5.xlarge,
2 nodes min      c5.large — diversified)
(system pods)    2-10 nodes, auto-scaled
$0.048/hr        $0.010-0.015/hr (70% cheaper)
    |               |
    +------+--------+
           |
    Cluster Autoscaler
    (adds/removes Spot nodes
     based on pending pods)
           |
    Node Termination Handler
    (DaemonSet — catches Spot
     interruption warnings,
     drains node gracefully)

Problem Solved

A typical EKS cluster with 10 On-Demand m5.large nodes costs roughly $700/month. The same workload on a mix of 2 On-Demand m5.large (for system pods) and 8 Spot nodes costs around $200/month — a 70% reduction. For a startup spending $7,000/month on compute, this saves $4,900 every month.

The challenge with Spot is reliability — AWS gives only a 2-minute warning before reclaiming an instance. Without proper handling, pods on that node are killed immediately, causing failed requests and data corruption. The Node Termination Handler (NTH) catches the interruption notice and triggers a graceful kubectl drain — all pods are evicted to healthy nodes before AWS terminates the instance. From the application's perspective, it looks like a planned maintenance event, not a crash.

Step-by-Step Implementation Guide

Step 1: Create the EKS Cluster with Mixed Node Groups

HCL

## eks-spot.tf — main infrastructure
 
## On-Demand node group for system workloads
resource "aws_eks_node_group" "on_demand" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "on-demand-system"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = aws_subnet.private[*].id
  capacity_type   = "ON_DEMAND"  # Reliable, not interruptible
  instance_types  = ["t3.medium"]
 
  scaling_config {
    desired_size = 2
    min_size     = 2
    max_size     = 4
  }
 
  labels = {
    "node-type" = "on-demand"
    "workload"  = "system"
  }
 
  taint {
    key    = "system-only"
    value  = "true"
    effect = "NO_SCHEDULE"  # Prevent application pods from landing here
  }
 
  tags = {
    # These tags are REQUIRED for Cluster Autoscaler to manage this group
    "k8s.io/cluster-autoscaler/enabled"                  = "true"
    "k8s.io/cluster-autoscaler/${var.cluster_name}"      = "owned"
  }
}
 
## Spot node group for application workloads
resource "aws_eks_node_group" "spot" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "spot-applications"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = aws_subnet.private[*].id
  capacity_type   = "SPOT"
  # CRITICAL: Use multiple instance types for Spot — if one is unavailable,
  # EKS automatically tries the next one
  # This is called instance type diversification and prevents Spot shortages
  instance_types  = [
    "m5.large",    # Primary choice
    "m5a.large",   # AMD variant — usually cheaper, same performance
    "m4.large",    # Previous generation — often available when m5 is not
    "c5.xlarge",   # More CPU, same memory cost
    "r5.large",    # More memory option
  ]
 
  scaling_config {
    desired_size = 2
    min_size     = 1   # At least 1 node always running
    max_size     = 10  # Scale up to 10 nodes during traffic spikes
  }
 
  labels = {
    "node-type"  = "spot"
    "workload"   = "application"
  }
 
  tags = {
    "k8s.io/cluster-autoscaler/enabled"                  = "true"
    "k8s.io/cluster-autoscaler/${var.cluster_name}"      = "owned"
    "k8s.io/cluster-autoscaler/node-template/label/node-type" = "spot"
  }
 
  depends_on = [
    aws_iam_role_policy_attachment.node_AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.node_AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.node_AmazonEC2ContainerRegistryReadOnly,
  ]
}

Step 2: Install and Configure Cluster Autoscaler

Bash

## Create the IAM policy for Cluster Autoscaler
## It needs permission to describe and modify Auto Scaling Groups
cat > cluster-autoscaler-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "autoscaling:DescribeTags",
        "ec2:DescribeImages",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:GetInstanceTypesFromInstanceRequirements",
        "eks:DescribeNodegroup"
      ],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": ["*"]
    }
  ]
}
EOF
 
## Create the IAM role with IRSA
eksctl create iamserviceaccount \
  --cluster=YOUR_CLUSTER_NAME \
  --namespace=kube-system \
  --name=cluster-autoscaler \
  --attach-policy-arn=arn:aws:iam::ACCOUNT_ID:policy/ClusterAutoscalerPolicy \
  --approve
 
## Install Cluster Autoscaler via Helm
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm repo update
 
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set autoDiscovery.clusterName=YOUR_CLUSTER_NAME \
  --set awsRegion=ap-south-1 \
  --set rbac.serviceAccount.create=false \
  --set rbac.serviceAccount.name=cluster-autoscaler \
  --set extraArgs.balance-similar-node-groups=true \
  --set extraArgs.skip-nodes-with-local-storage=false \
  --set extraArgs.expander=least-waste \
  --set extraArgs.scale-down-delay-after-add=5m \
  --set extraArgs.scale-down-unneeded-time=5m
 
## Verify Cluster Autoscaler is running
kubectl get pods -n kube-system | grep cluster-autoscaler
 
## Watch autoscaler logs to confirm it is watching node groups
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler \
  --tail=50 | grep -i "detected"

Step 3: Install Node Termination Handler

Bash

## The NTH runs as a DaemonSet on every node
## It watches for Spot interruption notices from AWS EC2 metadata service
## When a notice arrives, it cordons the node and evicts all pods
## This gives your pods 90+ seconds to reschedule elsewhere gracefully
 
helm repo add eks https://aws.github.io/eks-charts
helm repo update
 
helm install aws-node-termination-handler eks/aws-node-termination-handler \
  --namespace kube-system \
  --set enableSpotInterruptionDraining=true \
  --set enableRebalanceMonitoring=true \
  --set enableRebalanceDraining=true \
  --set enableScheduledEventDraining=true \
  --set podTerminationGracePeriod=120  # Give pods 2 minutes to shut down
 
## Verify NTH is running on all nodes (including Spot nodes)
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-node-termination-handler
## Expected: One pod per node (DaemonSet)

Step 4: Configure Pod Disruption Budgets

Bash

## Pod Disruption Budgets (PDBs) tell Kubernetes the minimum availability
## during voluntary disruptions like node draining.
## This prevents NTH from evicting too many pods at once.
 
kubectl apply -f - <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: webapp-pdb
spec:
  minAvailable: 2  # At least 2 webapp pods must be running during drain
  selector:
    matchLabels:
      app: webapp
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-service-pdb
spec:
  minAvailable: "50%"  # At least 50% of payment pods must be running
  selector:
    matchLabels:
      app: payment-service
EOF
 
## Verify PDBs are configured
kubectl get pdb

Step 5: Test Autoscaling and Spot Handling

Bash

## Test 1: Scale up by creating pending pods
## Deploy a resource-hungry deployment that requires more nodes
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scale-test
spec:
  replicas: 20  # More pods than current capacity
  selector:
    matchLabels:
      app: scale-test
  template:
    metadata:
      labels:
        app: scale-test
    spec:
      containers:
      * name: stress
        image: busybox
        command: ["sh", "-c", "sleep 600"]
        resources:
          requests:
            cpu: 500m     # Each pod needs 0.5 CPU
            memory: 512Mi # Each pod needs 512MB RAM
EOF
 
## Watch Cluster Autoscaler add new Spot nodes
kubectl get nodes --watch
## Expected: New nodes appear within 3-5 minutes
 
## Watch autoscaler logs for scale-up decision
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler \
  --tail=20 | grep -i scale
 
## Test 2: Scale down
kubectl delete deployment scale-test
## After 5 minutes (scale-down-unneeded-time), idle nodes are removed
kubectl get nodes --watch
 
## Test 3: Simulate Spot interruption (no actual interruption — just test the drain)
SPOT_NODE=$(kubectl get nodes -l node-type=spot -o jsonpath='{.items[0].metadata.name}')
kubectl drain $SPOT_NODE --ignore-daemonsets --delete-emptydir-data
## Expected: All pods evict gracefully to other nodes
## Verify no application errors during the drain:
kubectl get pods -o wide | grep $SPOT_NODE
## Expected: No pods remaining on the drained node

Validation & Testing

Bash

## 1. Verify node group configuration
aws eks describe-nodegroup \
  --cluster-name YOUR_CLUSTER_NAME \
  --nodegroup-name spot-applications \
  --query 'nodegroup.[capacityType,instanceTypes,scalingConfig]'
## Expected: capacityType=SPOT, multiple instance types, min/max/desired
 
## 2. Check current node costs
kubectl get nodes -o wide
## Check AWS console Cost Explorer for actual spend comparison
 
## 3. Verify Cluster Autoscaler is watching your node groups
kubectl logs -n kube-system deployment/cluster-autoscaler \
  | grep "Found.*node group"
## Expected: Shows both on-demand and spot node groups detected
 
## 4. Verify Node Termination Handler
kubectl get daemonset aws-node-termination-handler -n kube-system
## Expected: DESIRED and CURRENT match number of nodes
 
## 5. Verify Pod Disruption Budgets
kubectl get pdb -A
## Expected: webapp-pdb and payment-service-pdb with correct minAvailable
 
## 6. Cost comparison calculation
ON_DEMAND_NODES=$(kubectl get nodes -l node-type=on-demand --no-headers | wc -l)
SPOT_NODES=$(kubectl get nodes -l node-type=spot --no-headers | wc -l)
echo "On-Demand nodes: $ON_DEMAND_NODES (at ~$0.048/hr each)"
echo "Spot nodes: $SPOT_NODES (at ~$0.014/hr each)"
ON_DEMAND_COST=$(echo "$ON_DEMAND_NODES * 0.048 * 24 * 30" | bc)
SPOT_COST=$(echo "$SPOT_NODES * 0.014 * 24 * 30" | bc)
TOTAL=$((ON_DEMAND_COST + SPOT_COST))
FULL_ON_DEMAND=$(echo "$(($ON_DEMAND_NODES + $SPOT_NODES)) * 0.048 * 24 * 30" | bc)
echo "Monthly cost with Spot mix: $$TOTAL"
echo "Monthly cost with all On-Demand: $$FULL_ON_DEMAND"
echo "Monthly savings: $(($FULL_ON_DEMAND - $TOTAL))"

Videos & Guides

Video Guide

Kubernetes Cluster Autoscaler with Spot Instances — AWS Tutorial

Complete tutorial covering EKS Cluster Autoscaler setup, Spot instance node groups, Node Termination Handler installation, and Pod Disruption Budget configuration for production cost optimisation.

View Reference

DOCUMENTATION

AWS Node Termination Handler Documentation

Official AWS Node Termination Handler documentation — detecting Spot interruptions, graceful pod eviction, rebalance recommendations, and scheduled maintenance event handling.

View Reference

DOCUMENTATION

Kubernetes Cluster Autoscaler AWS Documentation

Official Cluster Autoscaler AWS provider documentation covering IAM configuration, node group tags, expander strategies, and advanced tuning parameters.

View Reference