Taint

Kubernetes hub

A key-value property applied to a Kubernetes node that repels pods from being scheduled onto it unless the pod explicitly declares a matching Toleration, enabling controlled and dedicated workload placement.

Taint — Extended Technical Detail

What is a Taint in Simple Terms?

A taint is like putting a "No Entry" sign on a node. By default, no pod will be scheduled on a tainted node. Only pods that explicitly say "I am allowed to go here" (via a Toleration) will be scheduled there.

◈ DIAGRAM

+------------------------------------------+
| Without Taint                            |
|                                          |
| gpu-node-1  <- any pod can land here    | <- Regular API pods, batch jobs,
| gpu-node-2  <- any pod can land here    |    and GPU workloads all compete
|                                          |    for expensive GPU nodes
+------------------------------------------+
 
+------------------------------------------+
| With Taint: workload=gpu:NoSchedule      |
|                                          |
| gpu-node-1  [TAINTED]                   | <- Only pods with matching
| gpu-node-2  [TAINTED]                   |    Toleration can land here.
|                                          |    API pods are repelled.
+------------------------------------------+

Real Use Case

At Hotstar, GPU nodes for video transcoding are expensive (10x the cost of regular nodes). Tainting them ensures only transcoding pods run there — not regular API pods accidentally consuming GPU capacity during a traffic spike.

Taint Effects — The Three Modes

◈ DIAGRAM

+------------------------+    +------------------------+    +------------------------+
| NoSchedule             |    | PreferNoSchedule       |    | NoExecute              |
|                        |    |                        |    |                        |
| New pods without       |    | Kubernetes will TRY    |    | New pods repelled AND  |
| Toleration will NOT    |    | to avoid this node     |    | existing pods without  |
| be scheduled here      |    | but may use it if no   |    | Toleration are EVICTED |
|                        |    | other node is free     |    |                        |
| Existing pods: safe    |    | Existing pods: safe    |    | Use for: maintenance   |
| Use for: isolation     |    | Use for: soft hints    |    | and node draining      |
+------------------------+    +------------------------+    +------------------------+

How to Add a Taint to a Node

Bash

# Taint a node — only GPU workloads allowed
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule
# Taint format: key=value:effect
 
# Taint for node maintenance — evicts non-tolerating pods immediately
kubectl taint nodes mumbai-worker-3 maintenance=patching:NoExecute
 
# Taint without a value (key-only taint)
kubectl taint nodes gpu-node-1 dedicated:NoSchedule
 
# Verify the taint was applied
kubectl describe node gpu-node-1 | grep -A 5 Taints
# Taints: workload=gpu:NoSchedule

How to Remove a Taint

Bash

# Add a minus sign at the end to remove — exact key=value:effect match required
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule-
 
# Remove a key-only taint
kubectl taint nodes gpu-node-1 dedicated:NoSchedule-
 
# Verify taint is gone
kubectl describe node gpu-node-1 | grep Taints
# Taints: <none>

The Matching Toleration in Pod Spec

A taint does nothing without a matching Toleration in the pod spec:

YAML

# deployment.yaml — GPU transcoding pod that tolerates the taint
apiVersion: apps/v1
kind: Deployment
metadata:
  name: video-transcoder
  namespace: streaming-prod
spec:
  template:
    spec:
      tolerations:
        - key: "workload"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"    # Must match the taint effect exactly
      nodeSelector:
        workload: gpu             # Also add nodeSelector to PREFER GPU nodes
      containers:
        - name: transcoder
          image: registry.hotstar.in/transcoder:v4.2.1
          resources:
            limits:
              nvidia.com/gpu: "1"  # Request 1 GPU

How to View Taints on All Nodes

Bash

# Custom column output — quick overview
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
 
# Full taint details including key, value, effect
kubectl get nodes -o json | \
  jq '.items[] | {name: .metadata.name, taints: .spec.taints}'
 
# Check a specific node
kubectl describe node mumbai-worker-3 | grep -A 10 Taints

Taint + Toleration + NodeAffinity — The Complete Pattern

For hard node dedication (only GPU pods on GPU nodes, and GPU pods ONLY on GPU nodes), combine all three:

YAML

spec:
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: workload
                operator: In
                values: ["gpu"]

◈ DIAGRAM

+------------------------------------------+
| Taint on node                            | <- Repels non-GPU pods
+------------------------------------------+
| Toleration on pod                        | <- Allows GPU pod past the repel
+------------------------------------------+
| NodeAffinity on pod                      | <- Ensures GPU pod goes ONLY to
|                                          |    GPU nodes (not just tolerates)
+------------------------------------------+

Troubleshooting Common Taint Problems

Problem	Symptom	Fix
Pod stuck in `Pending` with taint	`0/5 nodes are available: 5 node(s) had taint`	Pod missing Toleration — add matching `tolerations` block to pod spec
Toleration set but pod still Pending	Pod has Toleration but won't schedule	Toleration `effect` doesn't match taint `effect` — they must be identical
Non-GPU pods landing on GPU nodes	Cost spike on GPU instance billing	Taint is set but pods have a catch-all Toleration (`operator: Exists`) — scope the Toleration to a specific key and value
NoExecute evicting critical pods	Running pods evicted unexpectedly	Wrong node tainted — verify `kubectl get nodes` names before applying `NoExecute`
Taint not removed after maintenance	Node still repelling pods after patch	Forgot the trailing `-` in `kubectl taint` remove command — re-run with `-` suffix

Remember
Taints and Tolerations work together. A taint on a node does nothing unless the pod also has the matching Toleration. Forgetting the Toleration means the pod stays in Pending forever — and kubectl describe pod will show node(s) had taint in the Events section.

Tip
Use NoExecute taint effect during node maintenance. It automatically evicts all non-tolerating pods, so you can safely patch the node without running kubectl drain manually. The system daemonsets (kube-proxy, CNI) always have built-in Tolerations for system taints and will not be evicted.

Security
On PCI-DSS workloads like Razorpay's card processing pods, use NoSchedule taints on dedicated nodes to ensure no other workload can co-reside on those nodes. Co-tenancy on the same node means potential side-channel attacks via shared CPU caches — taint-based isolation is a critical compliance control.

Common Mistake
Using operator: Exists without a key in a Toleration. This creates a wildcard Toleration that matches ALL taints on ALL nodes — effectively defeating every taint in the cluster for that pod. Always scope Tolerations to a specific key.

Frequently Asked Questions

How is a Kubernetes taint different from just using node selectors or affinity rules?

Node affinity and selectors are pull-based — a pod declares which nodes it wants. A taint is push-based — it's applied to the node and repels all pods by default, regardless of what the pod specifies, unless the pod explicitly declares a matching Toleration. This inversion matters for dedicating nodes to specific workloads (like GPU nodes) where you want to guarantee general-purpose pods stay off, not just hope they don't get scheduled there.

What's a common mistake with taints and tolerations in production clusters?

Confusing tolerations with affinity — a toleration only allows a pod to be scheduled on a tainted node, it doesn't attract it there, so without also adding a nodeAffinity or nodeSelector, your pod might tolerate the taint but still land on a completely different, untainted node. Also, the `NoExecute` taint effect actively evicts already-running pods that lack a matching toleration, which surprises engineers who only tested with `NoSchedule` (which merely blocks new scheduling).