Overview and What You Will Learn
Regular Deployments treat every pod as identical and interchangeable — perfect for stateless APIs but catastrophic for databases where pod identity, startup order, and storage persistence are critical. StatefulSets solve this by giving each pod a stable, predictable identity, its own dedicated PersistentVolumeClaim, and strict ordered deployment and termination guarantees. This lab walks you through deploying PostgreSQL, Redis, and a multi-node database cluster on Kubernetes using StatefulSets with production-grade configuration.
By the end of this guide you will be able to:
- Understand the core differences between Deployments and StatefulSets and when to use each
- Deploy a single-instance PostgreSQL database using a StatefulSet with persistent storage
- Configure a Redis cluster using StatefulSets with stable network identities
- Set up a primary-replica PostgreSQL configuration with ordered pod startup
- Troubleshoot common StatefulSet failures including stuck termination and PVC binding issues
Why This Matters in Production
Zerodha runs PostgreSQL for trade records and MySQL for user accounts directly on Kubernetes using StatefulSets. The ordered startup guarantee means the primary database pod always initialises and becomes ready before replica pods attempt to connect and begin replication — preventing the split-brain scenarios that plague manually managed database clusters.
At Razorpay, Redis is deployed as a StatefulSet cluster where each node has a stable DNS name (redis-0.redis, redis-1.redis, redis-2.redis) that never changes even after pod restarts. Application code hardcodes these stable names rather than dynamic pod IPs — impossible with a regular Deployment.
Core Principles
StatefulSet vs Deployment — the critical differences: DEPLOYMENT STATEFULSET ────────── ─────────── Pod names Random suffix Stable ordinal api-7d9f8b-xkp2q postgres-0 api-7d9f8b-mn3lp postgres-1 postgres-2 Pod identity Interchangeable Unique and stable Storage Shared or none Each pod gets its own dedicated PVC (postgres-data-0, postgres-data-1) Startup order All pods start Ordered: pod-0 must simultaneously be Ready before pod-1 starts Termination All pods stop Reverse order: simultaneously pod-2 → pod-1 → pod-0 DNS Service IP only Per-pod DNS: pod-0.service.ns.svc.cluster.local
When to use StatefulSet vs Deployment: Use StatefulSet when: Use Deployment when: ──────────────────── ──────────────────
Databases (PostgreSQL, MySQL) * REST APIs Message queues (Kafka, RabbitMQ) * Web servers (NGINX, Express) Caches with persistence (Redis) * Background workers (stateless) Search engines (Elasticsearch) * Any app with no local state Any app needing stable pod DNS * Any app that is truly stateless
Detailed Step-by-Step Practical Lab
Step 1 — Create the Headless Service for Stable Pod DNS
StatefulSets require a Headless Service — a Service with clusterIP: None that creates individual DNS entries for each pod instead of a single load-balanced IP:
apiVersion: v1kind: Servicemetadata: name: postgres namespace: production labels: app: postgresspec: clusterIP: None # This makes it a Headless Service selector: app: postgres ports: - name: postgres port: 5432 targetPort: 5432kubectl apply -f headless-service-postgres.yaml # This creates DNS entries for each pod:# postgres-0.postgres.production.svc.cluster.local → pod IP of postgres-0# postgres-1.postgres.production.svc.cluster.local → pod IP of postgres-1# postgres-2.postgres.production.svc.cluster.local → pod IP of postgres-2 # Also create a regular Service for client connections (load balances reads)kubectl apply -f - <<EOFapiVersion: v1kind: Servicemetadata: name: postgres-primary namespace: productionspec: selector: app: postgres role: primary # Only route to the primary pod ports: - port: 5432 targetPort: 5432EOFRememberThe Headless Service name must match the
serviceNamefield in your StatefulSet spec — this is what enables the stable per-pod DNS names. Getting this wrong is the most common StatefulSet configuration mistake.
Step 2 — Deploy Single-Instance PostgreSQL StatefulSet
# statefulset-postgres.yaml — production PostgreSQL on KubernetesapiVersion: apps/v1kind: StatefulSetmetadata: name: postgres namespace: productionspec: serviceName: "postgres" # Must match the Headless Service name replicas: 1 # Start single — add replicas for HA selector: matchLabels: app: postgres template: metadata: labels: app: postgres role: primary spec: terminationGracePeriodSeconds: 60 # Give PostgreSQL time to flush WAL securityContext: fsGroup: 999 # postgres UID — sets volume ownership runAsUser: 999 runAsNonRoot: true initContainers: # Fix permissions on the data directory before PostgreSQL starts - name: fix-permissions image: busybox:1.35 command: ["sh", "-c", "chown -R 999:999 /var/lib/postgresql/data"] volumeMounts: - name: postgres-data mountPath: /var/lib/postgresql/data securityContext: runAsUser: 0 # Run as root for chown only containers: - name: postgres image: postgres:15.4 ports: - containerPort: 5432 name: postgres env: - name: POSTGRES_DB value: "zerodha_trading" - name: POSTGRES_USER valueFrom: secretKeyRef: name: postgres-credentials key: username - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: postgres-credentials key: password - name: PGDATA value: "/var/lib/postgresql/data/pgdata" # Subdirectory avoids lost+found - name: POSTGRES_INITDB_ARGS value: "--encoding=UTF8 --auth-host=scram-sha-256" resources: requests: cpu: "500m" memory: "1Gi" limits: cpu: "4" memory: "8Gi" livenessProbe: exec: command: - pg_isready - -U - $(POSTGRES_USER) - -d - $(POSTGRES_DB) initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 6 readinessProbe: exec: command: - pg_isready - -U - $(POSTGRES_USER) - -d - $(POSTGRES_DB) initialDelaySeconds: 5 periodSeconds: 5 failureThreshold: 3 volumeMounts: - name: postgres-data mountPath: /var/lib/postgresql/data - name: postgres-config mountPath: /etc/postgresql/postgresql.conf subPath: postgresql.conf volumeClaimTemplates: # Each pod gets its own PVC automatically - metadata: name: postgres-data labels: app: postgres spec: accessModes: ["ReadWriteOnce"] storageClassName: gp3-encrypted resources: requests: storage: 100Gikubectl apply -f statefulset-postgres.yaml # Watch ordered pod startupkubectl get pods -n production -w# NAME READY STATUS RESTARTS# postgres-0 0/1 ContainerCreating 0 ← starts first# postgres-0 0/1 Running 0# postgres-0 1/1 Running 0 ← must be Ready before replicas start # Verify PVC was automatically createdkubectl get pvc -n production# NAME STATUS VOLUME CAPACITY# postgres-data-postgres-0 Bound pvc-a1b2c3d4-... 100GiStep 3 — Deploy Redis as a StatefulSet Cluster
# statefulset-redis.yaml — Redis cluster with stable pod identitiesapiVersion: v1kind: ConfigMapmetadata: name: redis-config namespace: productiondata: redis.conf: | maxmemory 2gb maxmemory-policy allkeys-lru appendonly yes appendfsync everysec save 900 1 save 300 10 save 60 10000apiVersion: apps/v1kind: StatefulSetmetadata: name: redis namespace: productionspec: serviceName: "redis" replicas: 3 # 3-node Redis cluster selector: matchLabels: app: redis template: metadata: labels: app: redis spec: terminationGracePeriodSeconds: 30 containers: - name: redis image: redis:7.2 command: ["redis-server", "/etc/redis/redis.conf"] ports: - containerPort: 6379 name: redis resources: requests: cpu: "250m" memory: "512Mi" limits: cpu: "1" memory: "2Gi" livenessProbe: exec: command: ["redis-cli", "ping"] initialDelaySeconds: 15 periodSeconds: 10 readinessProbe: exec: command: ["redis-cli", "ping"] initialDelaySeconds: 5 periodSeconds: 5 volumeMounts: - name: redis-data mountPath: /data - name: redis-config mountPath: /etc/redis volumes: - name: redis-config configMap: name: redis-config volumeClaimTemplates: - metadata: name: redis-data spec: accessModes: ["ReadWriteOnce"] storageClassName: gp3-encrypted resources: requests: storage: 20Gikubectl apply -f statefulset-redis.yaml # Watch all 3 Redis pods start in strict orderkubectl get pods -n production -w# redis-0 1/1 Running 0 ← starts and becomes Ready first# redis-1 1/1 Running 0 ← starts only after redis-0 is Ready# redis-2 1/1 Running 0 ← starts only after redis-1 is Ready # Connect to Redis and verify clusterkubectl exec -it redis-0 -n production -- redis-cli ping# PONG # Each pod has a stable DNS name — application connects using these# redis-0.redis.production.svc.cluster.local:6379# redis-1.redis.production.svc.cluster.local:6379# redis-2.redis.production.svc.cluster.local:6379Step 4 — Perform a Rolling Update on a StatefulSet
# Update PostgreSQL image versionkubectl set image statefulset/postgres \ postgres=postgres:15.5 \ -n production # Watch ordered rolling update — updates in reverse order (pod-2 first, pod-0 last)kubectl rollout status statefulset/postgres -n production# Waiting for 1 pods to be ready...# statefulset rolling update complete 1 pods at revision postgres-6d8f9b... # Check rollout historykubectl rollout history statefulset/postgres -n production # Rollback if neededkubectl rollout undo statefulset/postgres -n productionTipStatefulSet rolling updates go in reverse ordinal order — pod-2 is updated first, then pod-1, then pod-0. For primary-replica databases this means replicas are updated before the primary, which is the safe order. Always verify replication lag is zero before each pod update completes.
Step 5 — Scale a StatefulSet Up and Down Safely
# Scale up — new pods start in order (pod-1 after pod-0 is Ready)kubectl scale statefulset postgres -n production --replicas=3 # Watch ordered scale-upkubectl get pods -n production -w# postgres-0 1/1 Running 0# postgres-1 0/1 Pending 0 ← starts after postgres-0 is Ready# postgres-1 1/1 Running 0# postgres-2 0/1 Pending 0 ← starts after postgres-1 is Ready# postgres-2 1/1 Running 0 # Scale down — pods terminate in reverse order (pod-2 first)kubectl scale statefulset postgres -n production --replicas=1 # CRITICAL: Scaling down does NOT delete PVCs# PVCs for postgres-1 and postgres-2 still exist after scale-downkubectl get pvc -n production | grep postgres# postgres-data-postgres-0 Bound 100Gi ← active# postgres-data-postgres-1 Bound 100Gi ← orphaned — delete manually if not needed# postgres-data-postgres-2 Bound 100Gi ← orphaned — delete manually if not neededSecurityNever delete orphaned PVCs automatically. Kubernetes intentionally keeps them to prevent accidental data loss. Review and manually delete them only after confirming the data is either replicated elsewhere or no longer needed.
Step 6 — Troubleshoot Common StatefulSet Failures
# Problem 1 — Pod stuck in Terminating statekubectl get pods -n production# postgres-0 1/1 Terminating 0 48m ← stuck # Cause: The pod has a finalizer or the node is unresponsive# Check for finalizerskubectl get pod postgres-0 -n production -o jsonpath='{.metadata.finalizers}' # Force delete as last resort (data loss risk — only if node is dead)kubectl delete pod postgres-0 -n production --force --grace-period=0 # Problem 2 — PVC stuck in Pending after scale-upkubectl describe pvc postgres-data-postgres-1 -n production# Events: ProvisioningFailed: no nodes available in zone ap-south-1a# Cause: WaitForFirstConsumer mode — pod must be scheduled first# Fix: Ensure the pod is scheduled before checking PVC status # Problem 3 — Pod-1 stuck in Init state waiting for pod-0kubectl get pods -n production# postgres-0 0/1 Running 0 ← not Ready yet (probe failing)# postgres-1 0/1 Init:0/1 0 ← waiting for postgres-0 to be Ready # Check why postgres-0 is not passing readiness probekubectl describe pod postgres-0 -n productionkubectl logs postgres-0 -n productionProduction Best Practices & Common Pitfalls
- Always set
terminationGracePeriodSecondsto at least 60 for databases. The default 30 seconds is too short for PostgreSQL to complete a checkpoint and flush WAL — abrupt termination risks data corruption. - Use
podManagementPolicy: Parallelonly for StatefulSets where pods are truly independent — like Elasticsearch data nodes. Never use it for primary-replica databases where order matters. - Monitor replication lag on all replica pods. A replica that falls too far behind the primary will cause data loss if the primary fails before the replica catches up.
- Back up PVCs using Velero with volume snapshots on a schedule — at minimum daily, ideally every hour for financial transaction databases.
- Use
updateStrategy: RollingUpdatewithpartitionduring major database version upgrades — this lets you upgrade one pod at a time and pause the rollout to verify replication before continuing.
Common MistakeDeleting a StatefulSet with
kubectl delete statefulset postgresthinking it will also clean up PVCs. It does not — PVCs are intentionally orphaned. But the pods are deleted, leaving your database inaccessible until the StatefulSet is recreated and the pods rebind to the orphaned PVCs. Always scale to zero first, verify, then delete.
Quick Reference & Troubleshooting Commands
| Command | Purpose |
|---|---|
kubectl get statefulset -n <ns> |
List all StatefulSets and replica counts |
kubectl describe statefulset <name> -n <ns> |
Full StatefulSet config and events |
kubectl get pods -n <ns> -w |
Watch ordered pod startup and termination |
kubectl scale statefulset <name> --replicas=<n> -n <ns> |
Scale StatefulSet up or down |
kubectl rollout status statefulset <name> -n <ns> |
Watch rolling update progress |
kubectl rollout undo statefulset <name> -n <ns> |
Rollback to previous StatefulSet revision |
kubectl exec -it <name>-0 -n <ns> -- bash |
Shell into the primary pod (ordinal 0) |
kubectl get pvc -n <ns> | grep <statefulset-name> |
List PVCs created by a StatefulSet |
kubectl delete pod <name>-0 -n <ns> --force --grace-period=0 |
Force delete stuck Terminating pod |
kubectl get pod <name>-0 -n <ns> -o jsonpath='{.metadata.finalizers}' |
Check for blocking finalizers |
