This project solves one of the most common complaints from DevOps engineers — Kubernetes costs too much. The solution is Spot instances combined with the Cluster Autoscaler. Spot instances are unused AWS capacity sold at up to 90% discount. The catch is AWS can reclaim them with a 2-minute warning. The Node Termination Handler catches that warning and gracefully drains the node before AWS takes it. This is the cost architecture used by Swiggy and Meesho — stateless application workloads run on Spot (cheap), stateful workloads and system components run on On-Demand (reliable). EKS Control Plane (managed by AWS) | +------+--------+ | | On-Demand Spot Instance Node Group Node Group (t3.medium) (m5.large, m5.xlarge, 2 nodes min c5.large — diversified) (system pods) 2-10 nodes, auto-scaled $0.048/hr $0.010-0.015/hr (70% cheaper) | | +------+--------+ | Cluster Autoscaler (adds/removes Spot nodes based on pending pods) | Node Termination Handler (DaemonSet — catches Spot interruption warnings, drains node gracefully)
A typical EKS cluster with 10 On-Demand m5.large nodes costs roughly $700/month. The same workload on a mix of 2 On-Demand m5.large (for system pods) and 8 Spot nodes costs around $200/month — a 70% reduction. For a startup spending $7,000/month on compute, this saves $4,900 every month. The challenge with Spot is reliability — AWS gives only a 2-minute warning before reclaiming an instance. Without proper handling, pods on that node are killed immediately, causing failed requests and data corruption. The Node Termination Handler (NTH) catches the interruption notice and triggers a graceful `kubectl drain` — all pods are evicted to healthy nodes before AWS terminates the instance. From the application's perspective, it looks like a planned maintenance event, not a crash.
### Step 1: Create the EKS Cluster with Mixed Node Groups ```hcl ## eks-spot.tf — main infrastructure ## On-Demand node group for system workloads resource "aws_eks_node_group" "on_demand" { cluster_name = aws_eks_cluster.main.name node_group_name = "on-demand-system" node_role_arn = aws_iam_role.node.arn subnet_ids = aws_subnet.private[*].id capacity_type = "ON_DEMAND" # Reliable, not interruptible instance_types = ["t3.medium"] scaling_config { desired_size = 2 min_size = 2 max_size = 4 } labels = { "node-type" = "on-demand" "workload" = "system" } taint { key = "system-only" value = "true" effect = "NO_SCHEDULE" # Prevent application pods from landing here } tags = { # These tags are REQUIRED for Cluster Autoscaler to manage this group "k8s.io/cluster-autoscaler/enabled" = "true" "k8s.io/cluster-autoscaler/${var.cluster_name}" = "owned" } } ## Spot node group for application workloads resource "aws_eks_node_group" "spot" { cluster_name = aws_eks_cluster.main.name node_group_name = "spot-applications" node_role_arn = aws_iam_role.node.arn subnet_ids = aws_subnet.private[*].id capacity_type = "SPOT" # CRITICAL: Use multiple instance types for Spot — if one is unavailable, # EKS automatically tries the next one # This is called instance type diversification and prevents Spot shortages instance_types = [ "m5.large", # Primary choice "m5a.large", # AMD variant — usually cheaper, same performance "m4.large", # Previous generation — often available when m5 is not "c5.xlarge", # More CPU, same memory cost "r5.large", # More memory option ] scaling_config { desired_size = 2 min_size = 1 # At least 1 node always running max_size = 10 # Scale up to 10 nodes during traffic spikes } labels = { "node-type" = "spot" "workload" = "application" } tags = { "k8s.io/cluster-autoscaler/enabled" = "true" "k8s.io/cluster-autoscaler/${var.cluster_name}" = "owned" "k8s.io/cluster-autoscaler/node-template/label/node-type" = "spot" } depends_on = [ aws_iam_role_policy_attachment.node_AmazonEKSWorkerNodePolicy, aws_iam_role_policy_attachment.node_AmazonEKS_CNI_Policy, aws_iam_role_policy_attachment.node_AmazonEC2ContainerRegistryReadOnly, ] } ``` ### Step 2: Install and Configure Cluster Autoscaler ```bash ## Create the IAM policy for Cluster Autoscaler ## It needs permission to describe and modify Auto Scaling Groups cat > cluster-autoscaler-policy.json << 'EOF' { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeAutoScalingInstances", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeScalingActivities", "autoscaling:DescribeTags", "ec2:DescribeImages", "ec2:DescribeInstanceTypes", "ec2:DescribeLaunchTemplateVersions", "ec2:GetInstanceTypesFromInstanceRequirements", "eks:DescribeNodegroup" ], "Resource": ["*"] }, { "Effect": "Allow", "Action": [ "autoscaling:SetDesiredCapacity", "autoscaling:TerminateInstanceInAutoScalingGroup" ], "Resource": ["*"] } ] } EOF ## Create the IAM role with IRSA eksctl create iamserviceaccount \ --cluster=YOUR_CLUSTER_NAME \ --namespace=kube-system \ --name=cluster-autoscaler \ --attach-policy-arn=arn:aws:iam::ACCOUNT_ID:policy/ClusterAutoscalerPolicy \ --approve ## Install Cluster Autoscaler via Helm helm repo add autoscaler https://kubernetes.github.io/autoscaler helm repo update helm install cluster-autoscaler autoscaler/cluster-autoscaler \ --namespace kube-system \ --set autoDiscovery.clusterName=YOUR_CLUSTER_NAME \ --set awsRegion=ap-south-1 \ --set rbac.serviceAccount.create=false \ --set rbac.serviceAccount.name=cluster-autoscaler \ --set extraArgs.balance-similar-node-groups=true \ --set extraArgs.skip-nodes-with-local-storage=false \ --set extraArgs.expander=least-waste \ --set extraArgs.scale-down-delay-after-add=5m \ --set extraArgs.scale-down-unneeded-time=5m ## Verify Cluster Autoscaler is running kubectl get pods -n kube-system | grep cluster-autoscaler ## Watch autoscaler logs to confirm it is watching node groups kubectl logs -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler \ --tail=50 | grep -i "detected" ``` ### Step 3: Install Node Termination Handler ```bash ## The NTH runs as a DaemonSet on every node ## It watches for Spot interruption notices from AWS EC2 metadata service ## When a notice arrives, it cordons the node and evicts all pods ## This gives your pods 90+ seconds to reschedule elsewhere gracefully helm repo add eks https://aws.github.io/eks-charts helm repo update helm install aws-node-termination-handler eks/aws-node-termination-handler \ --namespace kube-system \ --set enableSpotInterruptionDraining=true \ --set enableRebalanceMonitoring=true \ --set enableRebalanceDraining=true \ --set enableScheduledEventDraining=true \ --set podTerminationGracePeriod=120 # Give pods 2 minutes to shut down ## Verify NTH is running on all nodes (including Spot nodes) kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-node-termination-handler ## Expected: One pod per node (DaemonSet) ``` ### Step 4: Configure Pod Disruption Budgets ```bash ## Pod Disruption Budgets (PDBs) tell Kubernetes the minimum availability ## during voluntary disruptions like node draining. ## This prevents NTH from evicting too many pods at once. kubectl apply -f - <<EOF apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: webapp-pdb spec: minAvailable: 2 # At least 2 webapp pods must be running during drain selector: matchLabels: app: webapp --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: payment-service-pdb spec: minAvailable: "50%" # At least 50% of payment pods must be running selector: matchLabels: app: payment-service EOF ## Verify PDBs are configured kubectl get pdb ``` ### Step 5: Test Autoscaling and Spot Handling ```bash ## Test 1: Scale up by creating pending pods ## Deploy a resource-hungry deployment that requires more nodes kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: scale-test spec: replicas: 20 # More pods than current capacity selector: matchLabels: app: scale-test template: metadata: labels: app: scale-test spec: containers: * name: stress image: busybox command: ["sh", "-c", "sleep 600"] resources: requests: cpu: 500m # Each pod needs 0.5 CPU memory: 512Mi # Each pod needs 512MB RAM EOF ## Watch Cluster Autoscaler add new Spot nodes kubectl get nodes --watch ## Expected: New nodes appear within 3-5 minutes ## Watch autoscaler logs for scale-up decision kubectl logs -n kube-system -l app.kubernetes.io/name=aws-cluster-autoscaler \ --tail=20 | grep -i scale ## Test 2: Scale down kubectl delete deployment scale-test ## After 5 minutes (scale-down-unneeded-time), idle nodes are removed kubectl get nodes --watch ## Test 3: Simulate Spot interruption (no actual interruption — just test the drain) SPOT_NODE=$(kubectl get nodes -l node-type=spot -o jsonpath='{.items[0].metadata.name}') kubectl drain $SPOT_NODE --ignore-daemonsets --delete-emptydir-data ## Expected: All pods evict gracefully to other nodes ## Verify no application errors during the drain: kubectl get pods -o wide | grep $SPOT_NODE ## Expected: No pods remaining on the drained node ```
```bash ## 1. Verify node group configuration aws eks describe-nodegroup \ --cluster-name YOUR_CLUSTER_NAME \ --nodegroup-name spot-applications \ --query 'nodegroup.[capacityType,instanceTypes,scalingConfig]' ## Expected: capacityType=SPOT, multiple instance types, min/max/desired ## 2. Check current node costs kubectl get nodes -o wide ## Check AWS console Cost Explorer for actual spend comparison ## 3. Verify Cluster Autoscaler is watching your node groups kubectl logs -n kube-system deployment/cluster-autoscaler \ | grep "Found.*node group" ## Expected: Shows both on-demand and spot node groups detected ## 4. Verify Node Termination Handler kubectl get daemonset aws-node-termination-handler -n kube-system ## Expected: DESIRED and CURRENT match number of nodes ## 5. Verify Pod Disruption Budgets kubectl get pdb -A ## Expected: webapp-pdb and payment-service-pdb with correct minAvailable ## 6. Cost comparison calculation ON_DEMAND_NODES=$(kubectl get nodes -l node-type=on-demand --no-headers | wc -l) SPOT_NODES=$(kubectl get nodes -l node-type=spot --no-headers | wc -l) echo "On-Demand nodes: $ON_DEMAND_NODES (at ~$0.048/hr each)" echo "Spot nodes: $SPOT_NODES (at ~$0.014/hr each)" ON_DEMAND_COST=$(echo "$ON_DEMAND_NODES * 0.048 * 24 * 30" | bc) SPOT_COST=$(echo "$SPOT_NODES * 0.014 * 24 * 30" | bc) TOTAL=$((ON_DEMAND_COST + SPOT_COST)) FULL_ON_DEMAND=$(echo "$(($ON_DEMAND_NODES + $SPOT_NODES)) * 0.048 * 24 * 30" | bc) echo "Monthly cost with Spot mix: $$TOTAL" echo "Monthly cost with all On-Demand: $$FULL_ON_DEMAND" echo "Monthly savings: $(($FULL_ON_DEMAND - $TOTAL))" ```
This project solves one of the most common complaints from DevOps engineers — Kubernetes costs too much. The solution is...
A typical EKS cluster with 10 On-Demand m5.large nodes costs roughly $700/month. The same workload on a mix of 2 On-Dema...
Step 1: Create the EKS Cluster with Mixed Node Groups Step 2: Install and Configure Cluster Autoscaler Step 3: Install N...
...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.