StatefulSet

Kubernetes hub

A Kubernetes workload controller designed for applications that require stable, persistent identity across pod restarts — such as databases, message queues, and distributed stores. Unlike Deployments, each StatefulSet pod gets a predictable name, dedicated storage, and starts/stops in a guaranteed sequential order.

StatefulSet — Running Stateful Apps in Kubernetes

Why Deployments Break for Databases

A standard Deployment treats all pods as identical and disposable. Pod names are random (postgres-7d9f8c-xkqzp), storage is not guaranteed to follow a pod, and all pods can start or die simultaneously.

For a PostgreSQL primary-replica setup at Zerodha, this is catastrophic:

◈ DIAGRAM

+------------------------------------------+
| Deployment (WRONG for databases)         |
|                                          |
| postgres-7d9f8c-xkqzp  <- random name   | <- If pod dies, new pod gets
| postgres-7d9f8c-ab1cd  <- random name   |    a different random name.
|                                          |    Replicas lose their target.
| Shared PVC: postgres-data                | <- Two pods writing to the same
|   ^ both pods mount this                |    disk = data corruption
+------------------------------------------+
 
+------------------------------------------+
| StatefulSet (CORRECT for databases)      |
|                                          |
| postgres-0  <- stable, permanent name   | <- Always the primary.
| postgres-1  <- stable, permanent name   |    Replicas always know where
| postgres-2  <- stable, permanent name   |    to replicate from.
|                                          |
| postgres-data-0  <- dedicated PVC       | <- Each pod owns its own disk.
| postgres-data-1  <- dedicated PVC       |    PVC survives pod restarts.
| postgres-data-2  <- dedicated PVC       |
+------------------------------------------+

The Three Guarantees StatefulSet Provides

1. Stable Network Identity

Each pod gets a permanent DNS name that never changes, even after restarts:

◈ DIAGRAM

+------------------------------------------+
| DNS record format:                       |
|                                          |
| <pod-name>.<headless-svc>.<ns>.svc.cluster.local
|                                          |
| postgres-0.postgres-svc.production.svc.cluster.local
| postgres-1.postgres-svc.production.svc.cluster.local
| postgres-2.postgres-svc.production.svc.cluster.local
+------------------------------------------+

2. Ordered Startup and Shutdown

◈ DIAGRAM

+------------+     +------------+     +------------+
| postgres-0 |     | postgres-1 |     | postgres-2 |
|            |     |            |     |            |
| Starts 1st | --> | Starts 2nd | --> | Starts 3rd |
| (primary)  |     | (Running?) |     | (Running?) |
+------------+     +------------+     +------------+
 
Shutdown order is reversed:
postgres-2 stops -> postgres-1 stops -> postgres-0 stops (primary last)

3. Persistent Volume Per Pod

Each pod gets its own PVC via volumeClaimTemplates. The PVC survives pod deletion — when the pod recreates, it reattaches to the same volume automatically.

A Real StatefulSet Manifest — Redis Cluster

YAML

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: production
spec:
  serviceName: redis-headless           # Must reference a Headless Service
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7.2-alpine
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: redis-data
              mountPath: /data          # Each pod mounts its own /data volume
          resources:
            requests:
              memory: "256Mi"
              cpu: "100m"
            limits:
              memory: "512Mi"
              cpu: "500m"
  volumeClaimTemplates:                 # This is what makes it a StatefulSet
    - metadata:
        name: redis-data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3-encrypted
        resources:
          requests:
            storage: 10Gi              # Each of 3 pods gets its own 10Gi PVC
                                       # Total: 30Gi provisioned automatically

Headless Service — The Required Companion

StatefulSets require a Headless Service (clusterIP: None) to assign stable DNS records to each individual pod:

YAML

apiVersion: v1
kind: Service
metadata:
  name: redis-headless
  namespace: production
spec:
  clusterIP: None          # "Headless" — no virtual IP, just DNS per pod
  selector:
    app: redis
  ports:
    - port: 6379
      name: redis

◈ DIAGRAM

+------------------------------------------+    +------------------------------------------+
| Regular Service (clusterIP: 10.96.0.5)  |    | Headless Service (clusterIP: None)       |
|                                          |    |                                          |
| DNS: redis-svc.production.svc...         |    | DNS: redis-0.redis-headless.production.. |
| Routes to ANY redis pod (random)         |    | DNS: redis-1.redis-headless.production.. |
| Good for stateless read balancing        |    | Routes to SPECIFIC pod (by ordinal)      |
|                                          |    | Required for primary-replica topology    |
+------------------------------------------+    +------------------------------------------+

Remember
Without the Headless Service, pods don't get their individual DNS records. The serviceName in your StatefulSet spec MUST exactly match the Headless Service metadata.name — a mismatch means pods start but DNS never resolves.

Managing StatefulSet Pods

Bash

# List all pods — notice the stable ordinal names
kubectl get pods -n production -l app=redis
# NAME      READY   STATUS    RESTARTS   AGE
# redis-0   1/1     Running   0          5d
# redis-1   1/1     Running   0          5d
# redis-2   1/1     Running   0          5d
 
# List the auto-created PVCs (one per pod)
kubectl get pvc -n production -l app=redis
# NAME               STATUS   VOLUME                     CAPACITY
# redis-data-redis-0 Bound    pvc-3a8f2c1d-...           10Gi
# redis-data-redis-1 Bound    pvc-5c2e4b1a-...           10Gi
# redis-data-redis-2 Bound    pvc-7d3f9c2b-...           10Gi
 
# Scale up — redis-3 will be created with its own PVC
kubectl scale statefulset redis --replicas=4 -n production
 
# Rolling update (in reverse order: 2, 1, 0)
kubectl rollout status statefulset/redis -n production

StatefulSet vs Deployment — Decision Guide

Factor	Use Deployment	Use StatefulSet
App stores state to disk	No	Yes
Pods need unique identity	No	Yes
All pods are interchangeable	Yes	No
Startup order matters	No	Yes
Examples	API servers, web frontends	PostgreSQL, Kafka, Elasticsearch, Redis Cluster, ZooKeeper

Troubleshooting Common StatefulSet Problems

Problem	Symptom	Fix
Pod stuck in `Pending`	`redis-1` never starts	`redis-0` is not yet in `Running` state — fix the first pod before others proceed
PVC not binding	`redis-data-redis-0` stuck in `Pending`	StorageClass missing or wrong AZ — check `kubectl describe pvc redis-data-redis-0` events
Pod recreated but lost data	Data empty after restart	Pod reattached to wrong PVC — verify PVC name matches pod ordinal exactly
Headless Service DNS not resolving	`nslookup redis-0.redis-headless` fails	`serviceName` in StatefulSet spec doesn't match the Headless Service name
Scale-down left orphan PVCs	PVCs remain after `replicas: 1`	StatefulSet never auto-deletes PVCs on scale-down — delete manually after confirming data is safe

Common Mistake
Running a database as a Deployment with a single shared PVC. When the pod reschedules to a different node, the ReadWriteOnce PVC cannot follow to a different node, leaving the pod permanently stuck in Pending.

Security
Never run StatefulSet pods as root inside the container. A pod breakout on a database StatefulSet with root access can corrupt data or exfiltrate the entire volume. Always set securityContext.runAsNonRoot: true and readOnlyRootFilesystem: true on StatefulSet containers.

Tip
When upgrading StatefulSets (e.g., Redis version bump), use updateStrategy: RollingUpdate with partition: 2 to canary-test the upgrade on only the last pod first. Once verified healthy, set partition: 0 to roll out to all pods.

Frequently Asked Questions

Why can't I just run a database as a regular Kubernetes Deployment?

A Deployment treats pods as interchangeable — it assigns random suffixes to pod names, can recreate pods in any order, and by default all replicas share the same PersistentVolumeClaim template, meaning they'd all try to mount the same volume. A StatefulSet instead gives each pod a stable, predictable name (`app-0`, `app-1`, `app-2`), a dedicated PersistentVolumeClaim per pod that persists across rescheduling, and deploys/scales pods one at a time in order — properties databases and quorum-based systems like Kafka or etcd depend on.

What's a common StatefulSet mistake teams make in production?

Assuming StatefulSet gives you data replication or backup automatically — it doesn't; it only guarantees stable identity and storage, the replication logic (e.g., Postgres streaming replication, Kafka partition replicas) is still the application's job to configure. Also, scaling down a StatefulSet doesn't delete its PersistentVolumeClaims by default, which is usually what you want (data isn't lost) but surprises people who expect a clean teardown.