Cut EKS Costs with Spot and Cluster Autoscaler

Run EKS on mixed On-Demand and Spot node groups with Cluster Autoscaler, Node Termination Handler and PDBs, then measure your own savings.

Architecture Overview & Problem Statement

This project adds a Spot node group to an EKS cluster, runs stateless application pods on it, and uses Cluster Autoscaler to grow and shrink that capacity as pods become pending or idle. Pod Disruption Budgets and the AWS Node Termination Handler keep a Spot reclaim from looking like an outage. You finish by measuring what the change actually saved on your own bill, and what it cost in interruptions, which is the evidence to show in a portfolio.

You need a working EKS cluster, such as the one from Deploy a Production EKS Cluster with Terraform, with an OIDC provider and a node IAM role.

What you build:

An On-Demand node group for system pods and a Spot node group for applications
Cluster Autoscaler with a least-privilege IAM role
The Node Termination Handler
A sample workload pinned to Spot, with a Pod Disruption Budget
A savings calculation based on your real prices and node hours

flowchart TD CP[EKS control plane] --> OD[On-Demand node group
system pods] CP --> SP[Spot node group
application pods] CA[Cluster Autoscaler] -->|adds, removes| SP NTH[Termination Handler] -->|drains on notice| SP AWS[AWS reclaims capacity] -->|2 minute notice| NTH PDB[Pod Disruption Budget] -.->|limits evictions| SP classDef base fill:#326CE5,color:#fff classDef spot fill:#F97316,color:#fff classDef ctl fill:#8B5CF6,color:#fff classDef safe fill:#10B981,color:#fff classDef risk fill:#EF4444,color:#fff class CP,OD base class SP spot class CA,NTH ctl class PDB safe class AWS risk

The problem this solves

Spot instances are spare EC2 capacity that AWS sells at a discount, with the catch that AWS can take the instance back with about two minutes of warning. AWS advertises discounts of up to 90 percent compared with On-Demand. The real discount depends on the instance type, the availability zone, and the time, and it changes, so treat any single figure as a target to measure and not a promise.

Your saving is smaller than the Spot discount, because only the Spot-eligible part of the cluster gets it. System pods and anything stateful stay on On-Demand. The overall result is roughly:

TEXT

blended saving = share of node-hours on Spot x Spot discount

If 60 percent of your node-hours run on Spot at a 65 percent discount, the cluster saves about 39 percent, not 65. Milestone 6 has you calculate your own number.

Tip
Spot suits stateless, replicated workloads that restart quickly. A single-replica database or a long job that cannot checkpoint is a poor fit.

Milestone 1: Add On-Demand and Spot Node Groups

This milestone creates the two node groups. The key design choice for Spot is instance type diversification: if AWS runs short of one instance type in a zone, the group can use another, which lowers the chance of being interrupted or unable to scale.

Cluster Autoscaler assumes every node in a group has the same CPU and memory, because it uses one instance type as the template when it decides whether pending pods would fit. So diversify across types that are the same size, not across sizes. Here all five Spot types have 2 vCPUs and 8 GiB.

This code assumes the cluster, node IAM role, and private subnets already exist and are passed in as variables.

HCL

variable "cluster_name" {
  type = string
}
 
variable "node_role_arn" {
  type = string
}
 
variable "private_subnet_ids" {
  type = list(string)
}
 
resource "aws_eks_node_group" "on_demand" {
  cluster_name    = var.cluster_name
  node_group_name = "on-demand-system"
  node_role_arn   = var.node_role_arn
  subnet_ids      = var.private_subnet_ids
  capacity_type   = "ON_DEMAND"
  instance_types  = ["t3.medium"]
 
  scaling_config {
    desired_size = 2
    min_size     = 2
    max_size     = 3
  }
 
  labels = {
    workload = "system"
  }
 
  lifecycle {
    # Cluster Autoscaler changes the desired size, and Terraform must not undo it
    ignore_changes = [scaling_config[0].desired_size]
  }
}
 
resource "aws_eks_node_group" "spot" {
  cluster_name    = var.cluster_name
  node_group_name = "spot-applications"
  node_role_arn   = var.node_role_arn
  subnet_ids      = var.private_subnet_ids
  capacity_type   = "SPOT"
 
  # All the same size: 2 vCPU and 8 GiB
  instance_types = ["m5.large", "m5a.large", "m5d.large", "m5n.large", "m4.large"]
 
  scaling_config {
    desired_size = 2
    min_size     = 1
    max_size     = 10
  }
 
  labels = {
    workload = "application"
  }
 
  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
}

Bash

terraform apply
kubectl get nodes -L eks.amazonaws.com/capacityType
# Expected: two ON_DEMAND nodes and two SPOT nodes

EKS labels every node with eks.amazonaws.com/capacityType (ON_DEMAND or SPOT), so you do not need to add your own label to tell them apart. Managed node groups also tag their Auto Scaling groups so Cluster Autoscaler can discover them. Confirm with aws autoscaling describe-tags rather than adding those tags yourself.

Common Mistake
Letting Terraform own desired_size. Every terraform apply then resets the node count that Cluster Autoscaler chose, and your cluster flaps between sizes. Use ignore_changes on that field.

Milestone 2: Install Cluster Autoscaler

This milestone installs the component that adds nodes when pods cannot be scheduled and removes nodes that stay underused. It needs permission to resize Auto Scaling groups, which you grant through an IAM role for its service account (IRSA), limited to groups that belong to your cluster.

Bash

export CLUSTER=your-cluster-name
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
 
cat > cluster-autoscaler-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "autoscaling:DescribeTags",
        "ec2:DescribeImages",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:GetInstanceTypesFromInstanceRequirements",
        "eks:DescribeNodegroup"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/k8s.io/cluster-autoscaler/${CLUSTER}": "owned"
        }
      }
    }
  ]
}
EOF
 
aws iam create-policy \
  --policy-name ClusterAutoscalerPolicy \
  --policy-document file://cluster-autoscaler-policy.json
 
eksctl create iamserviceaccount \
  --cluster $CLUSTER \
  --namespace kube-system \
  --name cluster-autoscaler \
  --attach-policy-arn arn:aws:iam::$ACCOUNT_ID:policy/ClusterAutoscalerPolicy \
  --approve

The write permissions are limited by a tag condition, so a bug or compromise cannot resize Auto Scaling groups that belong to other clusters.

Create ca-values.yaml. The autoscaler's image version must match your cluster's Kubernetes minor version, so look that up first.

Bash

kubectl version
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm repo update
helm search repo autoscaler/cluster-autoscaler

YAML

autoDiscovery:
  clusterName: your-cluster-name
awsRegion: ap-south-1
 
rbac:
  serviceAccount:
    create: false
    name: cluster-autoscaler
 
image:
  tag: <version matching your Kubernetes minor version>
 
extraArgs:
  expander: least-waste
  scale-down-unneeded-time: 5m
  scale-down-delay-after-add: 5m
 
# Keep the autoscaler itself off Spot nodes
nodeSelector:
  eks.amazonaws.com/capacityType: ON_DEMAND

Bash

helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --values ca-values.yaml
 
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler --tail=30
# Expected: it lists both node groups and reports no errors

Security
Never give the autoscaler autoscaling:* on *. Resizing the wrong group, or terminating instances you did not intend, is an easy way to cause an outage. Keep the tag condition.

Milestone 3: Install the Node Termination Handler

This milestone installs the AWS Node Termination Handler, which watches the instance metadata service for interruption notices and drains the node before AWS takes it. The drain moves pods to healthy nodes, so a reclaim looks like planned maintenance and not a crash.

Bash

helm repo add eks https://aws.github.io/eks-charts
helm repo update
 
helm install aws-node-termination-handler eks/aws-node-termination-handler \
  --namespace kube-system \
  --set enableSpotInterruptionDraining=true \
  --set enableRebalanceMonitoring=true \
  --set enableRebalanceDraining=true \
  --set enableScheduledEventDraining=true
 
kubectl get daemonset -n kube-system -l app.kubernetes.io/name=aws-node-termination-handler
# Expected: one pod per node

Check the current EKS documentation before you rely on this. Managed node groups already react to Spot rebalance recommendations and interruption notices on their own, by launching a replacement and draining the old node. So with managed groups the handler overlaps with built-in behaviour. It matters most for self-managed groups and for scheduled maintenance events. Install it here so you understand the mechanism, then compare what each layer did during the test in Milestone 5.

Milestone 4: Deploy a Spot-Tolerant Workload

This milestone creates the workload you will test with. Nothing lands on Spot by chance, because the scheduler fills any node with room. You must tell the pods to run on Spot with a node affinity, and tell Kubernetes how much disruption they can take with a Pod Disruption Budget.

Bash

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 4
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      terminationGracePeriodSeconds: 30
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: eks.amazonaws.com/capacityType
                    operator: In
                    values: ["SPOT"]
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: webapp
      containers:
        - name: webapp
          image: nginx:stable
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: webapp
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: webapp
EOF
 
kubectl get pods -l app=webapp -o wide
kubectl get pdb

The topology spread asks the scheduler to put the replicas on different nodes, so one reclaimed node cannot remove them all. The PDB limits how many pods a voluntary drain may evict at once, which protects availability during Spot reclaims and during Cluster Autoscaler scale-down.

Common Mistake
Setting a PDB with minAvailable equal to the replica count. Then no pod can ever be evicted, and a drain hangs forever. Keep minAvailable below the number of replicas.

Milestone 5: Test Scale-Up, Scale-Down, and Interruption

This milestone proves each piece works, and records what happens, which you need for the measurement.

Scale up. Create more demand than the current nodes can hold and watch Cluster Autoscaler add Spot nodes:

Bash

kubectl scale deployment webapp --replicas=30
kubectl get pods -l app=webapp | grep -c Pending
kubectl get nodes -L eks.amazonaws.com/capacityType --watch
# Expected: pending pods, then new SPOT nodes within a few minutes
 
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler --tail=20 | grep -i scale

Scale down.

Bash

kubectl scale deployment webapp --replicas=4
kubectl get nodes --watch
# Expected: after about 5 minutes, idle Spot nodes are removed

Interruption handling. A real interruption cannot be triggered on demand, so simulate its effect with a drain, and watch the PDB protect you:

Bash

SPOT_NODE=$(kubectl get nodes -l eks.amazonaws.com/capacityType=SPOT -o jsonpath='{.items[0].metadata.name}')
kubectl drain $SPOT_NODE --ignore-daemonsets --delete-emptydir-data
kubectl get pods -l app=webapp -o wide
# Expected: pods move to other nodes and at least 2 stay available
 
kubectl uncordon $SPOT_NODE

While the drain runs, request the app in a loop from another terminal and count any failures. That count is one of the numbers you report.

For a more realistic test, AWS Fault Injection Service has an action that sends a real Spot interruption notice to chosen instances. Try it as an extension and compare what the node group, the handler, and your PDB each did.

Milestone 6: Measure Your Savings

This milestone replaces assumptions with your own data. Collect these inputs over at least a few days of normal running, and record them in your README:

The On-Demand price per hour for your instance types in ap-south-1, from the AWS pricing page
The Spot price you actually paid, from Cost Explorer filtered by purchase option, or sampled from the price history:

Bash

aws ec2 describe-spot-price-history \
  --region ap-south-1 \
  --instance-types m5.large \
  --product-descriptions "Linux/UNIX" \
  --start-time $(date -u -d '-7 days' +%Y-%m-%dT%H:%M:%SZ) \
  --query 'SpotPriceHistory[].[AvailabilityZone,SpotPrice,Timestamp]' \
  --output table

The node-hours you used on each capacity type, from Cost Explorer or from node counts over time

Then calculate the result with your own numbers. Edit the values at the top:

Python

od_price = 0.0     # On-Demand $/hour for the instance type
spot_price = 0.0   # average Spot $/hour you paid
od_nodes = 2       # average On-Demand nodes
spot_nodes = 4     # average Spot nodes
hours = 24 * 30
 
baseline = (od_nodes + spot_nodes) * od_price * hours
mixed = od_nodes * od_price * hours + spot_nodes * spot_price * hours
 
print(f"All On-Demand: ${baseline:,.2f}")
print(f"Mixed fleet:   ${mixed:,.2f}")
print(f"Saving:        ${baseline - mixed:,.2f} ({(baseline - mixed) / baseline:.0%})")

Fill in a results table in your README, with the target as a stated aim and the measured value as the result:

Measure	Target	Measured
Effective Spot discount on your instance types	As high as possible
Share of node-hours on Spot	At least 50%
Blended cluster saving	Positive, and explained
Spot interruptions in the test period	Recorded, not hidden
Failed requests during a drain	Zero or explained

Be honest about trade-offs. Spot capacity can be briefly unavailable, interruptions cost some risk, and the effort to run this has a cost. A saving that you measured and explained is worth more than a large one you cannot defend.

Validation & Testing

Confirm each part before you call the project finished.

Bash

# 1. Node groups have the right capacity types
aws eks describe-nodegroup --cluster-name $CLUSTER --nodegroup-name spot-applications \
  --query 'nodegroup.[capacityType,instanceTypes,scalingConfig]'
# Expected: SPOT, five 2 vCPU types, and the min and max you set
 
# 2. Cluster Autoscaler sees both groups, and has no errors
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler --tail=50
 
# 3. The termination handler runs on every node
kubectl get daemonset -n kube-system -l app.kubernetes.io/name=aws-node-termination-handler
 
# 4. The PDB exists and the workload runs only on Spot
kubectl get pdb
kubectl get pods -l app=webapp -o wide
 
# 5. Terraform shows no drift after the autoscaler resized the group
terraform plan
# Expected: no changes to desired_size

In your portfolio README, include the architecture diagram, the scale-up and scale-down evidence, the drain test result, and the completed savings table. Finish by listing what you would change before running this in production, such as a second Spot node group for another instance family, and a separate On-Demand group for stateful work.

Clean up when you finish: scale the workload to zero, delete the Helm releases, and destroy the node groups, so no Spot or On-Demand nodes keep billing.

Found a mistake, an outdated step or a broken link?

Cut EKS Costs with Spot and Cluster Autoscaler

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem this solves

Milestone 1: Add On-Demand and Spot Node Groups

Milestone 2: Install Cluster Autoscaler

Milestone 3: Install the Node Termination Handler

Milestone 4: Deploy a Spot-Tolerant Workload

Milestone 5: Test Scale-Up, Scale-Down, and Interruption

Milestone 6: Measure Your Savings

Validation & Testing