A Pod Disruption Budget (PDB) is a policy that tells Kubernetes the minimum number of pods that must stay running during voluntary disruptions — node drains, cluster upgrades, and rolling deployments. Without one, a node drain can terminate all pods of a service simultaneously, causing a complete outage.
+++
Configuring Pod Disruption Budgets for Zero-Downtime Upgrades
The Problem PDB Solves
Imagine your payments API has 3 pods spread across 3 nodes. A cluster upgrade requires draining all nodes one by one. Without a PDB, Kubernetes can drain Node 1, terminating its pod — that is fine, you still have 2. But it can immediately drain Node 2 next. Now you have 1 pod serving all production traffic. Then Node 3. Zero pods. Complete outage.
A PDB prevents this by telling the cluster: "Never let availability drop below 2 pods while you drain nodes."
WITHOUT PDB: WITH PDB (minAvailable: 2): Node drain sequence: Node drain sequence: Node-1 drained → 2 pods running Node-1 drained → 2 pods running ✓Node-2 drained → 1 pod running Node-2 drain attempt:Node-3 drained → 0 pods ← OUTAGE Kubernetes checks PDB 2 pods available = minimum met WAIT — cannot proceed New pod scheduled first 3 pods running again Node-2 drained → 2 pods ✓Voluntary vs Involuntary Disruptions
PDBs only apply to voluntary disruptions — actions an administrator or the cluster itself initiates intentionally.
| Type | Examples | PDB Applies? |
|---|---|---|
| Voluntary | kubectl drain, cluster upgrade, node scaling down, admin deletes pod |
✅ Yes |
| Involuntary | Node hardware failure, kernel panic, out-of-memory kill | ❌ No |
PDB cannot protect you from a node dying unexpectedly. It only governs intentional operations.
Two Ways to Define a PDB
Option 1 — minAvailable: At least this many pods must be running at all times.
apiVersion: policy/v1kind: PodDisruptionBudgetmetadata: name: payments-api-pdb namespace: productionspec: minAvailable: 2 # At least 2 pods must be available during any disruption selector: matchLabels: app: payments-api # Targets pods with this labelOption 2 — maxUnavailable: At most this many pods can be down at the same time.
apiVersion: policy/v1kind: PodDisruptionBudgetmetadata: name: payments-api-pdb namespace: productionspec: maxUnavailable: 1 # Only 1 pod can be unavailable at a time selector: matchLabels: app: payments-api`minAvailable` vs `maxUnavailable` — Which to Use
Deployment: 5 replicas minAvailable: 3→ Kubernetes can disrupt at most 2 pods at a time→ Absolute number — stays fixed even if you scale the deployment maxUnavailable: 1→ Kubernetes can disrupt at most 1 pod at a time→ Percentage option: maxUnavailable: "20%" adjusts as replicas scale| Setting | Best For | Watch Out |
|---|---|---|
minAvailable: N |
Critical services where you know the exact floor (e.g. "always 2 payment pods") | If replicas drop below N for any reason, node drains will block indefinitely |
maxUnavailable: N |
Services where you want proportional safety as replicas scale | Less intuitive for ops teams to reason about in an incident |
maxUnavailable: "10%" |
Large deployments (20+ replicas) | Rounds down — 10% of 5 pods = 0, meaning nothing can be disrupted |
Critical mistake: Setting
minAvailableequal to your replica count. Example: 3 replicas withminAvailable: 3. Kubernetes can never drain a node because draining any node would violate the budget. Cluster upgrades will stall permanently until someone deletes the PDB.
Using Percentages
spec: minAvailable: "60%" # At least 60% of matched pods must be available # With 10 replicas: at least 6 must be running → up to 4 can be disrupted# With 5 replicas: at least 3 must be running → up to 2 can be disrupted# With 3 replicas: at least 2 must be running → up to 1 can be disruptedPercentages are useful for autoscaled deployments where replica count fluctuates — your PDB stays proportionally correct without manual updates.
Checking PDB Status
# List all PDBs in a namespacekubectl get pdb -n production # Output:# NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE# payments-api-pdb 2 N/A 1 5d# auth-service-pdb N/A 1 1 5d # "ALLOWED DISRUPTIONS" = how many pods can currently be taken down# If this is 0, node drains will block # Describe for full detailskubectl describe pdb payments-api-pdb -n productionWhy a Node Drain Gets Stuck
The most common scenario at Razorpay or Hotstar: a cluster upgrade is running, and one node refuses to drain. The drain command hangs. The reason is almost always a PDB with ALLOWED DISRUPTIONS: 0.
# Drain a node during cluster upgradekubectl drain node mumbai-worker-3 \ --ignore-daemonsets \ --delete-emptydir-data # Output when PDB is blocking:# error when evicting pods/"payments-api-7d9f8b-xk2p9" -n "production"# (will retry after 5s): Cannot evict pod as it would violate# the pod's disruption budget.# Diagnose why ALLOWED DISRUPTIONS is 0kubectl get pdb payments-api-pdb -n production # Then check the actual pod count vs minAvailablekubectl get pods -l app=payments-api -n production # If only 2 pods are running and minAvailable is 2:# No pod can be evicted — evicting any one drops below the minimum# Fix: Scale up the deployment to 3+ replicas first, then drainkubectl scale deployment payments-api --replicas=4 -n productionFull Production Setup — Deployment + PDB Together
# deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: payments-api namespace: productionspec: replicas: 3 selector: matchLabels: app: payments-api strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 # Rolling update can take down 1 pod at a time maxSurge: 1 # Can temporarily create 1 extra pod during rollout template: metadata: labels: app: payments-api # ← Must match the PDB selector exactly spec: containers: - name: api image: registry.razorpay.in/payments-api:v2.5.1 # pdb.yaml — Apply this alongside the DeploymentapiVersion: policy/v1kind: PodDisruptionBudgetmetadata: name: payments-api-pdb namespace: productionspec: maxUnavailable: 1 selector: matchLabels: app: payments-api # ← Must match the Deployment pod labels exactly# Apply both togetherkubectl apply -f deployment.yaml -f pdb.yaml -n production # Verify the PDB is correctly targeting podskubectl get pdb payments-api-pdb -n production# ALLOWED DISRUPTIONS should be 1 if 3 pods are running and maxUnavailable is 1PDB for StatefulSets
StatefulSets (databases, Kafka, Zookeeper) are particularly important to protect because they have no load balancer in front — each pod is individually addressable and a quorum may be required.
# zookeeper-pdb.yamlapiVersion: policy/v1kind: PodDisruptionBudgetmetadata: name: zookeeper-pdb namespace: productionspec: minAvailable: 2 # Zookeeper 3-node cluster: must keep 2 for quorum selector: matchLabels: app: zookeeper3-node Zookeeper cluster requires quorum of 2 to elect a leader.minAvailable: 2 ensures Kubernetes never drains 2 nodes simultaneously,which would break quorum and make the cluster read-only.Checking PDB During Cluster Upgrade (Incident Workflow)
# 1. Before draining, check all PDB statuses across the clusterkubectl get pdb --all-namespaces # 2. Identify any PDB with ALLOWED DISRUPTIONS = 0kubectl get pdb --all-namespaces | grep " 0 " # 3. For each blocking PDB, check actual pod countkubectl get pods -n <namespace> -l <label-from-pdb-selector> # 4. If pods are fewer than minAvailable due to earlier failures:kubectl scale deployment <name> --replicas=<higher-count> -n <namespace> # 5. Then retry the drainkubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data🔴 Common Mistake: Creating a PDB but using a selector that does not match any pods. The PDB exists but protects nothing. Always verify kubectl get pdb shows a non-zero ALLOWED DISRUPTIONS value after creation — if it shows N/A or the pod count is 0, your selector is wrong.
💡 Tip: In clusters running on EKS or GKE where the cloud provider performs node upgrades automatically, PDBs are your last line of defense against upgrade-caused outages. The cloud upgrade process respects PDBs before draining nodes. At Hotstar scale, every Deployment with more than 1 replica should have a PDB — with maxUnavailable: 1 as the safe default for stateless services.
