What is Toleration? | DevOps Dictionary

Toleration — Extended Technical Detail

What is a Toleration in Simple Terms?

A Toleration is the pod's way of saying "I am allowed on that restricted node." It is the matching key to the Taint lock on the node. Without a Toleration, a pod is automatically repelled by any node that carries a matching Taint.

◈ DIAGRAM

+------------------------------------------+
| Node: gpu-node-1                         |
| Taint: workload=gpu:NoSchedule           | <- Node repels all pods by default
+------------------------------------------+
 
+------------------------------------------+
| Regular API Pod                          |
| No Toleration                            | <- REPELLED — stays on untainted nodes
+------------------------------------------+
 
+------------------------------------------+
| GPU Transcoding Pod                      |
| Toleration: workload=gpu:NoSchedule      | <- ALLOWED — can land on tainted node
+------------------------------------------+

Toleration in a Pod Spec

YAML

1# Pod spec for a GPU encoding job that tolerates the GPU node taint
2apiVersion: v1
3kind: Pod
4metadata:
5  name: video-encoder
6  namespace: streaming-prod
7spec:
8  tolerations:
9    - key: "workload"
10      operator: "Equal"
11      value: "gpu"
12      effect: "NoSchedule"   # Must match the taint effect exactly
13  containers:
14    - name: encoder
15      image: registry.hotstar.in/video-encoder:v3.1.0
16      resources:
17        limits:
18          nvidia.com/gpu: 1

Toleration Operators — Equal vs Exists

◈ DIAGRAM

+------------------------+          +------------------------------+
| operator: Equal        |          | operator: Exists             |
|                        |          |                              |
| key=workload           | <------> | key=workload                 |
| value=gpu              |          | (value is ignored)           |
| effect=NoSchedule      |          | effect=NoSchedule            |
|                        |          |                              |
| Matches ONLY taints    |          | Matches ANY taint with       |
| where key=workload     |          | key=workload regardless      |
| AND value=gpu          |          | of its value                 |
+------------------------+          +------------------------------+

Bash

1# Equal — key AND value must both match the taint exactly
2tolerations:
3  - key: "workload"
4    operator: "Equal"
5    value: "gpu"
6    effect: "NoSchedule"
7 
8# Exists — only the key needs to match, value is ignored
9tolerations:
10  - key: "workload"
11    operator: "Exists"
12    effect: "NoSchedule"
13 
14# Universal — tolerates ALL taints on ALL nodes (empty key + Exists)
15tolerations:
16  - operator: "Exists"   # No key — matches every taint in the cluster

Toleration with tolerationSeconds (for NoExecute)

When a node gets a NoExecute taint (e.g., during a node failure), Kubernetes evicts pods that don't tolerate it. You can delay eviction using tolerationSeconds:

YAML

1spec:
2  tolerations:
3    - key: "node.kubernetes.io/not-ready"
4      operator: "Exists"
5      effect: "NoExecute"
6      tolerationSeconds: 300   # Stay on the node for 5 minutes before eviction
7                               # Useful for Zerodha's trading pods during brief node blips

◈ DIAGRAM

+------------------------------------------+
| Node becomes NotReady                    | <- Hardware hiccup on mumbai-worker-3
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| NoExecute taint auto-applied             | <- Kubernetes adds system taint
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Pod with tolerationSeconds: 300          | <- Waits 5 min before evicting
| Pod without tolerationSeconds            | <- Evicted immediately
+------------------------------------------+

Toleration + NodeAffinity — The Complete Pattern

A Toleration only allows a pod onto a tainted node — it does not guarantee the pod lands there. To ensure the pod goes exclusively to the intended node, combine Toleration with NodeAffinity:

YAML

1# deployment.yaml — video transcoder exclusively on GPU nodes at Hotstar
2spec:
3  tolerations:
4    - key: "workload"
5      operator: "Equal"
6      value: "gpu"
7      effect: "NoSchedule"       # Step 1: Allow past the taint repel
8  affinity:
9    nodeAffinity:
10      requiredDuringSchedulingIgnoredDuringExecution:
11        nodeSelectorTerms:
12          - matchExpressions:
13              - key: workload
14                operator: In
15                values: ["gpu"]  # Step 2: Pin to GPU nodes only
16  containers:
17    - name: transcoder
18      image: registry.hotstar.in/transcoder:v4.2.1
19      resources:
20        limits:
21          nvidia.com/gpu: "1"

◈ DIAGRAM

+------------------------------------------+
| Toleration on pod                        | <- Unlocks the tainted node
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| NodeAffinity on pod                      | <- Forces pod to GPU nodes only
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| GPU node exclusively runs GPU workloads  | <- Isolation achieved
+------------------------------------------+

System Tolerations on DaemonSets

Critical system DaemonSets (kube-proxy, CNI, log collectors) automatically carry built-in Tolerations so they run on every node including tainted ones:

Bash

1# Check the built-in tolerations on a system DaemonSet
2kubectl get daemonset kube-proxy -n kube-system -o yaml | grep -A 20 tolerations
3 
4# Output — kube-proxy tolerates everything
5# tolerations:
6# - operator: Exists    <-- matches ALL taints, runs everywhere

Troubleshooting Common Toleration Problems

Problem	Symptom	Fix
Pod stuck in Pending	`0/5 nodes available: node(s) had taint`	Pod is missing a Toleration — add `tolerations` block with matching key, value, and effect
Toleration set but pod still Pending	Pod has Toleration but won't schedule	Toleration `effect` doesn't match taint `effect` — they must be identical
Pod not landing on intended node	Pod schedules on random untainted nodes	Toleration allows the node but doesn't pin the pod — add `nodeAffinity` or `nodeSelector`
Non-GPU pods landing on GPU nodes	Cost spike on GPU instance billing	Pod has `operator: Exists` with no key — wildcard Toleration bypasses all taints
Pod evicted unexpectedly after node event	Running pod disappears during node hiccup	Missing `tolerationSeconds` on `NoExecute` Toleration — add a grace period

💡 Tip: Use operator: Exists with an empty key to create a pod that tolerates ALL taints on ALL nodes — useful for critical system DaemonSets like log collectors (Fluentd, Filebeat) that must run on every node including tainted GPU and dedicated nodes.

📌 Remember: A Toleration is permission, not a directive. It tells Kubernetes "this pod is allowed on tainted nodes" — but without a nodeAffinity or nodeSelector, the scheduler may still place the pod on a cheaper untainted node. Always pair Toleration with affinity rules for hard node dedication.

⚠️ Security: At Razorpay, PCI-DSS compliance requires that card-processing pods run only on dedicated, isolated nodes. Set operator: Equal with the exact taint key and value — never operator: Exists — to prevent these pods from accidentally tolerating unintended taints and landing on shared infrastructure.

🔴 Common Mistake: Using operator: Exists without a key in a Toleration. This is a wildcard that matches every taint on every node in the cluster — effectively defeating all taint-based isolation for that pod. Always scope Tolerations to a specific key unless you explicitly need universal placement (system DaemonSets only).

Syncing Data

Toleration

Technical Explanation & Usage

Toleration — Extended Technical Detail

What is a Toleration in Simple Terms?

Toleration in a Pod Spec

Toleration Operators — Equal vs Exists

Toleration with tolerationSeconds (for NoExecute)

Toleration + NodeAffinity — The Complete Pattern

System Tolerations on DaemonSets

Troubleshooting Common Toleration Problems

Related Terms

Namespace

Pod

Deployment