What is StatefulSet? | DevOps Dictionary

StatefulSet — Running Stateful Apps in Kubernetes

Why Deployments Break for Databases

A standard Deployment treats all pods as identical and disposable. Pod names are random (postgres-7d9f8c-xkqzp), storage is not guaranteed to follow a pod, and all pods can start or die simultaneously.

For a PostgreSQL primary-replica setup at Zerodha, this is catastrophic:

◈ DIAGRAM

+------------------------------------------+
| Deployment (WRONG for databases)         |
|                                          |
| postgres-7d9f8c-xkqzp  <- random name   | <- If pod dies, new pod gets
| postgres-7d9f8c-ab1cd  <- random name   |    a different random name.
|                                          |    Replicas lose their target.
| Shared PVC: postgres-data                | <- Two pods writing to the same
|   ^ both pods mount this                |    disk = data corruption
+------------------------------------------+
 
+------------------------------------------+
| StatefulSet (CORRECT for databases)      |
|                                          |
| postgres-0  <- stable, permanent name   | <- Always the primary.
| postgres-1  <- stable, permanent name   |    Replicas always know where
| postgres-2  <- stable, permanent name   |    to replicate from.
|                                          |
| postgres-data-0  <- dedicated PVC       | <- Each pod owns its own disk.
| postgres-data-1  <- dedicated PVC       |    PVC survives pod restarts.
| postgres-data-2  <- dedicated PVC       |
+------------------------------------------+

The Three Guarantees StatefulSet Provides

1. Stable Network Identity

Each pod gets a permanent DNS name that never changes, even after restarts:

◈ DIAGRAM

+------------------------------------------+
| DNS record format:                       |
|                                          |
| <pod-name>.<headless-svc>.<ns>.svc.cluster.local
|                                          |
| postgres-0.postgres-svc.production.svc.cluster.local
| postgres-1.postgres-svc.production.svc.cluster.local
| postgres-2.postgres-svc.production.svc.cluster.local
+------------------------------------------+

2. Ordered Startup and Shutdown

◈ DIAGRAM

+------------+     +------------+     +------------+
| postgres-0 |     | postgres-1 |     | postgres-2 |
|            |     |            |     |            |
| Starts 1st | --> | Starts 2nd | --> | Starts 3rd |
| (primary)  |     | (Running?) |     | (Running?) |
+------------+     +------------+     +------------+
 
Shutdown order is reversed:
postgres-2 stops -> postgres-1 stops -> postgres-0 stops (primary last)

3. Persistent Volume Per Pod

Each pod gets its own PVC via volumeClaimTemplates. The PVC survives pod deletion — when the pod recreates, it reattaches to the same volume automatically.

A Real StatefulSet Manifest — Redis Cluster

YAML

1apiVersion: apps/v1
2kind: StatefulSet
3metadata:
4  name: redis
5  namespace: production
6spec:
7  serviceName: redis-headless           # Must reference a Headless Service
8  replicas: 3
9  selector:
10    matchLabels:
11      app: redis
12  template:
13    metadata:
14      labels:
15        app: redis
16    spec:
17      containers:
18        - name: redis
19          image: redis:7.2-alpine
20          ports:
21            - containerPort: 6379
22          volumeMounts:
23            - name: redis-data
24              mountPath: /data          # Each pod mounts its own /data volume
25          resources:
26            requests:
27              memory: "256Mi"
28              cpu: "100m"
29            limits:
30              memory: "512Mi"
31              cpu: "500m"
32  volumeClaimTemplates:                 # This is what makes it a StatefulSet
33    - metadata:
34        name: redis-data
35      spec:
36        accessModes: ["ReadWriteOnce"]
37        storageClassName: gp3-encrypted
38        resources:
39          requests:
40            storage: 10Gi              # Each of 3 pods gets its own 10Gi PVC
41                                       # Total: 30Gi provisioned automatically

Headless Service — The Required Companion

StatefulSets require a Headless Service (clusterIP: None) to assign stable DNS records to each individual pod:

YAML

1apiVersion: v1
2kind: Service
3metadata:
4  name: redis-headless
5  namespace: production
6spec:
7  clusterIP: None          # "Headless" — no virtual IP, just DNS per pod
8  selector:
9    app: redis
10  ports:
11    - port: 6379
12      name: redis

◈ DIAGRAM

+------------------------------------------+    +------------------------------------------+
| Regular Service (clusterIP: 10.96.0.5)  |    | Headless Service (clusterIP: None)       |
|                                          |    |                                          |
| DNS: redis-svc.production.svc...         |    | DNS: redis-0.redis-headless.production.. |
| Routes to ANY redis pod (random)         |    | DNS: redis-1.redis-headless.production.. |
| Good for stateless read balancing        |    | Routes to SPECIFIC pod (by ordinal)      |
|                                          |    | Required for primary-replica topology    |
+------------------------------------------+    +------------------------------------------+

📌 Remember: Without the Headless Service, pods don't get their individual DNS records. The serviceName in your StatefulSet spec MUST exactly match the Headless Service metadata.name — a mismatch means pods start but DNS never resolves.

Managing StatefulSet Pods

Bash

1# List all pods — notice the stable ordinal names
2kubectl get pods -n production -l app=redis
3# NAME      READY   STATUS    RESTARTS   AGE
4# redis-0   1/1     Running   0          5d
5# redis-1   1/1     Running   0          5d
6# redis-2   1/1     Running   0          5d
7 
8# List the auto-created PVCs (one per pod)
9kubectl get pvc -n production -l app=redis
10# NAME               STATUS   VOLUME                     CAPACITY
11# redis-data-redis-0 Bound    pvc-3a8f2c1d-...           10Gi
12# redis-data-redis-1 Bound    pvc-5c2e4b1a-...           10Gi
13# redis-data-redis-2 Bound    pvc-7d3f9c2b-...           10Gi
14 
15# Scale up — redis-3 will be created with its own PVC
16kubectl scale statefulset redis --replicas=4 -n production
17 
18# Rolling update (in reverse order: 2, 1, 0)
19kubectl rollout status statefulset/redis -n production

StatefulSet vs Deployment — Decision Guide

Factor	Use Deployment	Use StatefulSet
App stores state to disk	No	Yes
Pods need unique identity	No	Yes
All pods are interchangeable	Yes	No
Startup order matters	No	Yes
Examples	API servers, web frontends	PostgreSQL, Kafka, Elasticsearch, Redis Cluster, ZooKeeper

Troubleshooting Common StatefulSet Problems

Problem	Symptom	Fix
Pod stuck in `Pending`	`redis-1` never starts	`redis-0` is not yet in `Running` state — fix the first pod before others proceed
PVC not binding	`redis-data-redis-0` stuck in `Pending`	StorageClass missing or wrong AZ — check `kubectl describe pvc redis-data-redis-0` events
Pod recreated but lost data	Data empty after restart	Pod reattached to wrong PVC — verify PVC name matches pod ordinal exactly
Headless Service DNS not resolving	`nslookup redis-0.redis-headless` fails	`serviceName` in StatefulSet spec doesn't match the Headless Service name
Scale-down left orphan PVCs	PVCs remain after `replicas: 1`	StatefulSet never auto-deletes PVCs on scale-down — delete manually after confirming data is safe

🔴 Common Mistake: Running a database as a Deployment with a single shared PVC. When the pod reschedules to a different node, the ReadWriteOnce PVC cannot follow to a different node, leaving the pod permanently stuck in Pending.

⚠️ Security: Never run StatefulSet pods as root inside the container. A pod breakout on a database StatefulSet with root access can corrupt data or exfiltrate the entire volume. Always set securityContext.runAsNonRoot: true and readOnlyRootFilesystem: true on StatefulSet containers.

💡 Tip: When upgrading StatefulSets (e.g., Redis version bump), use updateStrategy: RollingUpdate with partition: 2 to canary-test the upgrade on only the last pod first. Once verified healthy, set partition: 0 to roll out to all pods.

Syncing Data

StatefulSet

Technical Explanation & Usage

StatefulSet — Running Stateful Apps in Kubernetes

Why Deployments Break for Databases

The Three Guarantees StatefulSet Provides

A Real StatefulSet Manifest — Redis Cluster

Headless Service — The Required Companion

Managing StatefulSet Pods

StatefulSet vs Deployment — Decision Guide

Troubleshooting Common StatefulSet Problems

Related Terms

Namespace

Pod

Deployment