What is the career path for learning Running StatefulSets for Databases on Kubernetes?

Mastering Running StatefulSets for Databases on Kubernetes enables engineering opportunities in DevOps, SRE, and cloud platform automation.

How long does it take to learn Running StatefulSets for Databases on Kubernetes?

Most students gain core proficiency in Running StatefulSets for Databases on Kubernetes in 2–3 weeks of active hands-on labs.

Running StatefulSets for Databases on Kubernetes | DevOps Network

Running StatefulSets for Databases on Kubernetes

Overview and What You Will Learn

Regular Deployments treat every pod as identical and interchangeable — perfect for stateless APIs but catastrophic for databases where pod identity, startup order, and storage persistence are critical. StatefulSets solve this by giving each pod a stable, predictable identity, its own dedicated PersistentVolumeClaim, and strict ordered deployment and termination guarantees. This lab walks you through deploying PostgreSQL, Redis, and a multi-node database cluster on Kubernetes using StatefulSets with production-grade configuration.

By the end of this guide you will be able to:

Understand the core differences between Deployments and StatefulSets and when to use each
Deploy a single-instance PostgreSQL database using a StatefulSet with persistent storage
Configure a Redis cluster using StatefulSets with stable network identities
Set up a primary-replica PostgreSQL configuration with ordered pod startup
Troubleshoot common StatefulSet failures including stuck termination and PVC binding issues

Why This Matters in Production

Zerodha runs PostgreSQL for trade records and MySQL for user accounts directly on Kubernetes using StatefulSets. The ordered startup guarantee means the primary database pod always initialises and becomes ready before replica pods attempt to connect and begin replication — preventing the split-brain scenarios that plague manually managed database clusters.

At Razorpay, Redis is deployed as a StatefulSet cluster where each node has a stable DNS name (redis-0.redis, redis-1.redis, redis-2.redis) that never changes even after pod restarts. Application code hardcodes these stable names rather than dynamic pod IPs — impossible with a regular Deployment.

Core Principles

StatefulSet vs Deployment — the critical differences: DEPLOYMENT STATEFULSET ────────── ─────────── Pod names Random suffix Stable ordinal api-7d9f8b-xkp2q postgres-0 api-7d9f8b-mn3lp postgres-1 postgres-2 Pod identity Interchangeable Unique and stable Storage Shared or none Each pod gets its own dedicated PVC (postgres-data-0, postgres-data-1) Startup order All pods start Ordered: pod-0 must simultaneously be Ready before pod-1 starts Termination All pods stop Reverse order: simultaneously pod-2 → pod-1 → pod-0 DNS Service IP only Per-pod DNS: pod-0.service.ns.svc.cluster.local

When to use StatefulSet vs Deployment: Use StatefulSet when: Use Deployment when: ──────────────────── ──────────────────

Databases (PostgreSQL, MySQL) * REST APIs Message queues (Kafka, RabbitMQ) * Web servers (NGINX, Express) Caches with persistence (Redis) * Background workers (stateless) Search engines (Elasticsearch) * Any app with no local state Any app needing stable pod DNS * Any app that is truly stateless

Detailed Step-by-Step Practical Lab

Step 1 — Create the Headless Service for Stable Pod DNS

StatefulSets require a Headless Service — a Service with clusterIP: None that creates individual DNS entries for each pod instead of a single load-balanced IP:

YAML

1# headless-service-postgres.yaml
2apiVersion: v1
3kind: Service
4metadata:
5  name: postgres
6  namespace: production
7  labels:
8    app: postgres
9spec:
10  clusterIP: None           # This makes it a Headless Service
11  selector:
12    app: postgres
13  ports:
14    - name: postgres
15      port: 5432
16      targetPort: 5432

Bash

1kubectl apply -f headless-service-postgres.yaml
2 
3# This creates DNS entries for each pod:
4# postgres-0.postgres.production.svc.cluster.local → pod IP of postgres-0
5# postgres-1.postgres.production.svc.cluster.local → pod IP of postgres-1
6# postgres-2.postgres.production.svc.cluster.local → pod IP of postgres-2
7 
8# Also create a regular Service for client connections (load balances reads)
9kubectl apply -f - <<EOF
10apiVersion: v1
11kind: Service
12metadata:
13  name: postgres-primary
14  namespace: production
15spec:
16  selector:
17    app: postgres
18    role: primary             # Only route to the primary pod
19  ports:
20    - port: 5432
21      targetPort: 5432
22EOF

📌 Remember: The Headless Service name must match the serviceName field in your StatefulSet spec — this is what enables the stable per-pod DNS names. Getting this wrong is the most common StatefulSet configuration mistake.

Step 2 — Deploy Single-Instance PostgreSQL StatefulSet

YAML

1# statefulset-postgres.yaml — production PostgreSQL on Kubernetes
2apiVersion: apps/v1
3kind: StatefulSet
4metadata:
5  name: postgres
6  namespace: production
7spec:
8  serviceName: "postgres"       # Must match the Headless Service name
9  replicas: 1                   # Start single — add replicas for HA
10  selector:
11    matchLabels:
12      app: postgres
13  template:
14    metadata:
15      labels:
16        app: postgres
17        role: primary
18    spec:
19      terminationGracePeriodSeconds: 60   # Give PostgreSQL time to flush WAL
20      securityContext:
21        fsGroup: 999                      # postgres UID — sets volume ownership
22        runAsUser: 999
23        runAsNonRoot: true
24      initContainers:
25        # Fix permissions on the data directory before PostgreSQL starts
26        - name: fix-permissions
27          image: busybox:1.35
28          command: ["sh", "-c", "chown -R 999:999 /var/lib/postgresql/data"]
29          volumeMounts:
30            - name: postgres-data
31              mountPath: /var/lib/postgresql/data
32          securityContext:
33            runAsUser: 0                  # Run as root for chown only
34      containers:
35        - name: postgres
36          image: postgres:15.4
37          ports:
38            - containerPort: 5432
39              name: postgres
40          env:
41            - name: POSTGRES_DB
42              value: "zerodha_trading"
43            - name: POSTGRES_USER
44              valueFrom:
45                secretKeyRef:
46                  name: postgres-credentials
47                  key: username
48            - name: POSTGRES_PASSWORD
49              valueFrom:
50                secretKeyRef:
51                  name: postgres-credentials
52                  key: password
53            - name: PGDATA
54              value: "/var/lib/postgresql/data/pgdata"   # Subdirectory avoids lost+found
55            - name: POSTGRES_INITDB_ARGS
56              value: "--encoding=UTF8 --auth-host=scram-sha-256"
57          resources:
58            requests:
59              cpu: "500m"
60              memory: "1Gi"
61            limits:
62              cpu: "4"
63              memory: "8Gi"
64          livenessProbe:
65            exec:
66              command:
67                - pg_isready
68                - -U
69                - $(POSTGRES_USER)
70                - -d
71                - $(POSTGRES_DB)
72            initialDelaySeconds: 30
73            periodSeconds: 10
74            failureThreshold: 6
75          readinessProbe:
76            exec:
77              command:
78                - pg_isready
79                - -U
80                - $(POSTGRES_USER)
81                - -d
82                - $(POSTGRES_DB)
83            initialDelaySeconds: 5
84            periodSeconds: 5
85            failureThreshold: 3
86          volumeMounts:
87            - name: postgres-data
88              mountPath: /var/lib/postgresql/data
89            - name: postgres-config
90              mountPath: /etc/postgresql/postgresql.conf
91              subPath: postgresql.conf
92  volumeClaimTemplates:             # Each pod gets its own PVC automatically
93    - metadata:
94        name: postgres-data
95        labels:
96          app: postgres
97      spec:
98        accessModes: ["ReadWriteOnce"]
99        storageClassName: gp3-encrypted
100        resources:
101          requests:
102            storage: 100Gi

Bash

1kubectl apply -f statefulset-postgres.yaml
2 
3# Watch ordered pod startup
4kubectl get pods -n production -w
5# NAME         READY   STATUS              RESTARTS
6# postgres-0   0/1     ContainerCreating   0        ← starts first
7# postgres-0   0/1     Running             0
8# postgres-0   1/1     Running             0        ← must be Ready before replicas start
9 
10# Verify PVC was automatically created
11kubectl get pvc -n production
12# NAME                    STATUS   VOLUME                    CAPACITY
13# postgres-data-postgres-0  Bound  pvc-a1b2c3d4-...          100Gi

Step 3 — Deploy Redis as a StatefulSet Cluster

YAML

1# statefulset-redis.yaml — Redis cluster with stable pod identities
2apiVersion: v1
3kind: ConfigMap
4metadata:
5  name: redis-config
6  namespace: production
7data:
8  redis.conf: |
9    maxmemory 2gb
10    maxmemory-policy allkeys-lru
11    appendonly yes
12    appendfsync everysec
13    save 900 1
14    save 300 10
15    save 60 10000
16apiVersion: apps/v1
17kind: StatefulSet
18metadata:
19  name: redis
20  namespace: production
21spec:
22  serviceName: "redis"
23  replicas: 3               # 3-node Redis cluster
24  selector:
25    matchLabels:
26      app: redis
27  template:
28    metadata:
29      labels:
30        app: redis
31    spec:
32      terminationGracePeriodSeconds: 30
33      containers:
34        - name: redis
35          image: redis:7.2
36          command: ["redis-server", "/etc/redis/redis.conf"]
37          ports:
38            - containerPort: 6379
39              name: redis
40          resources:
41            requests:
42              cpu: "250m"
43              memory: "512Mi"
44            limits:
45              cpu: "1"
46              memory: "2Gi"
47          livenessProbe:
48            exec:
49              command: ["redis-cli", "ping"]
50            initialDelaySeconds: 15
51            periodSeconds: 10
52          readinessProbe:
53            exec:
54              command: ["redis-cli", "ping"]
55            initialDelaySeconds: 5
56            periodSeconds: 5
57          volumeMounts:
58            - name: redis-data
59              mountPath: /data
60            - name: redis-config
61              mountPath: /etc/redis
62      volumes:
63        - name: redis-config
64          configMap:
65            name: redis-config
66  volumeClaimTemplates:
67    - metadata:
68        name: redis-data
69      spec:
70        accessModes: ["ReadWriteOnce"]
71        storageClassName: gp3-encrypted
72        resources:
73          requests:
74            storage: 20Gi

Bash

1kubectl apply -f statefulset-redis.yaml
2 
3# Watch all 3 Redis pods start in strict order
4kubectl get pods -n production -w
5# redis-0   1/1   Running   0    ← starts and becomes Ready first
6# redis-1   1/1   Running   0    ← starts only after redis-0 is Ready
7# redis-2   1/1   Running   0    ← starts only after redis-1 is Ready
8 
9# Connect to Redis and verify cluster
10kubectl exec -it redis-0 -n production -- redis-cli ping
11# PONG
12 
13# Each pod has a stable DNS name — application connects using these
14# redis-0.redis.production.svc.cluster.local:6379
15# redis-1.redis.production.svc.cluster.local:6379
16# redis-2.redis.production.svc.cluster.local:6379

Step 4 — Perform a Rolling Update on a StatefulSet

Bash

1# Update PostgreSQL image version
2kubectl set image statefulset/postgres \
3  postgres=postgres:15.5 \
4  -n production
5 
6# Watch ordered rolling update — updates in reverse order (pod-2 first, pod-0 last)
7kubectl rollout status statefulset/postgres -n production
8# Waiting for 1 pods to be ready...
9# statefulset rolling update complete 1 pods at revision postgres-6d8f9b...
10 
11# Check rollout history
12kubectl rollout history statefulset/postgres -n production
13 
14# Rollback if needed
15kubectl rollout undo statefulset/postgres -n production

💡 Tip: StatefulSet rolling updates go in reverse ordinal order — pod-2 is updated first, then pod-1, then pod-0. For primary-replica databases this means replicas are updated before the primary, which is the safe order. Always verify replication lag is zero before each pod update completes.

Step 5 — Scale a StatefulSet Up and Down Safely

Bash

1# Scale up — new pods start in order (pod-1 after pod-0 is Ready)
2kubectl scale statefulset postgres -n production --replicas=3
3 
4# Watch ordered scale-up
5kubectl get pods -n production -w
6# postgres-0   1/1   Running   0
7# postgres-1   0/1   Pending   0   ← starts after postgres-0 is Ready
8# postgres-1   1/1   Running   0
9# postgres-2   0/1   Pending   0   ← starts after postgres-1 is Ready
10# postgres-2   1/1   Running   0
11 
12# Scale down — pods terminate in reverse order (pod-2 first)
13kubectl scale statefulset postgres -n production --replicas=1
14 
15# CRITICAL: Scaling down does NOT delete PVCs
16# PVCs for postgres-1 and postgres-2 still exist after scale-down
17kubectl get pvc -n production | grep postgres
18# postgres-data-postgres-0   Bound   100Gi  ← active
19# postgres-data-postgres-1   Bound   100Gi  ← orphaned — delete manually if not needed
20# postgres-data-postgres-2   Bound   100Gi  ← orphaned — delete manually if not needed

⚠️ Security: Never delete orphaned PVCs automatically. Kubernetes intentionally keeps them to prevent accidental data loss. Review and manually delete them only after confirming the data is either replicated elsewhere or no longer needed.

Step 6 — Troubleshoot Common StatefulSet Failures

Bash

1# Problem 1 — Pod stuck in Terminating state
2kubectl get pods -n production
3# postgres-0   1/1   Terminating   0   48m  ← stuck
4 
5# Cause: The pod has a finalizer or the node is unresponsive
6# Check for finalizers
7kubectl get pod postgres-0 -n production -o jsonpath='{.metadata.finalizers}'
8 
9# Force delete as last resort (data loss risk — only if node is dead)
10kubectl delete pod postgres-0 -n production --force --grace-period=0
11 
12# Problem 2 — PVC stuck in Pending after scale-up
13kubectl describe pvc postgres-data-postgres-1 -n production
14# Events: ProvisioningFailed: no nodes available in zone ap-south-1a
15# Cause: WaitForFirstConsumer mode — pod must be scheduled first
16# Fix: Ensure the pod is scheduled before checking PVC status
17 
18# Problem 3 — Pod-1 stuck in Init state waiting for pod-0
19kubectl get pods -n production
20# postgres-0   0/1   Running   0   ← not Ready yet (probe failing)
21# postgres-1   0/1   Init:0/1  0   ← waiting for postgres-0 to be Ready
22 
23# Check why postgres-0 is not passing readiness probe
24kubectl describe pod postgres-0 -n production
25kubectl logs postgres-0 -n production

Production Best Practices & Common Pitfalls

Always set terminationGracePeriodSeconds to at least 60 for databases. The default 30 seconds is too short for PostgreSQL to complete a checkpoint and flush WAL — abrupt termination risks data corruption.
Use podManagementPolicy: Parallel only for StatefulSets where pods are truly independent — like Elasticsearch data nodes. Never use it for primary-replica databases where order matters.
Monitor replication lag on all replica pods. A replica that falls too far behind the primary will cause data loss if the primary fails before the replica catches up.
Back up PVCs using Velero with volume snapshots on a schedule — at minimum daily, ideally every hour for financial transaction databases.
Use updateStrategy: RollingUpdate with partition during major database version upgrades — this lets you upgrade one pod at a time and pause the rollout to verify replication before continuing.

🔴 Common Mistake: Deleting a StatefulSet with kubectl delete statefulset postgres thinking it will also clean up PVCs. It does not — PVCs are intentionally orphaned. But the pods are deleted, leaving your database inaccessible until the StatefulSet is recreated and the pods rebind to the orphaned PVCs. Always scale to zero first, verify, then delete.

Quick Reference & Troubleshooting Commands

Command	Purpose
`kubectl get statefulset -n <ns>`	List all StatefulSets and replica counts
`kubectl describe statefulset <name> -n <ns>`	Full StatefulSet config and events
`kubectl get pods -n <ns> -w`	Watch ordered pod startup and termination
`kubectl scale statefulset <name> --replicas=<n> -n <ns>`	Scale StatefulSet up or down
`kubectl rollout status statefulset <name> -n <ns>`	Watch rolling update progress
`kubectl rollout undo statefulset <name> -n <ns>`	Rollback to previous StatefulSet revision
`kubectl exec -it <name>-0 -n <ns> -- bash`	Shell into the primary pod (ordinal 0)
`kubectl get pvc -n <ns> \| grep <statefulset-name>`	List PVCs created by a StatefulSet
`kubectl delete pod <name>-0 -n <ns> --force --grace-period=0`	Force delete stuck Terminating pod
`kubectl get pod <name>-0 -n <ns> -o jsonpath='{.metadata.finalizers}'`	Check for blocking finalizers

Syncing Data

Running StatefulSets for Databases on Kubernetes

Running StatefulSets for Databases on Kubernetes

Overview and What You Will Learn

Why This Matters in Production

Core Principles

Detailed Step-by-Step Practical Lab

Step 1 — Create the Headless Service for Stable Pod DNS

Step 2 — Deploy Single-Instance PostgreSQL StatefulSet

Step 3 — Deploy Redis as a StatefulSet Cluster

Step 4 — Perform a Rolling Update on a StatefulSet

Step 5 — Scale a StatefulSet Up and Down Safely

Step 6 — Troubleshoot Common StatefulSet Failures

Production Best Practices & Common Pitfalls

Quick Reference & Troubleshooting Commands

Resources

Explore More in Kubernetes Workload Management

Troubleshooting Kubernetes Pod OOMKilled and CrashLoopBackOff Errors

Configuring Ingress Controllers with NGINX for Production Traffic

Managing Kubernetes Secrets with Vault and ConfigMaps

Scaling Deployments with Horizontal Pod Autoscaler (HPA)