etcd

Q: Why does Kubernetes use etcd specifically instead of a general-purpose database for cluster state?

etcd is built on the Raft consensus algorithm, giving it strong consistency guarantees across a cluster of etcd nodes (typically 3 or 5) even during network partitions or node failures — critical because the Kubernetes API server relies on etcd's data being correct, not eventually consistent, when scheduling decisions and controller reconciliation depend on it. Its watch API also lets controllers efficiently react to state changes in real time rather than polling.

Q: What's the most common operational mistake teams make with etcd in production clusters?

Not taking regular etcd snapshots (`etcdctl snapshot save`) and not testing restore procedures — etcd holds literally every piece of cluster state including Secrets, so losing quorum without a backup means rebuilding the entire cluster's configuration from scratch. Also common: running an even number of etcd members (e.g. 4) which doesn't improve fault tolerance over an odd number but does increase the chance of split votes; always run odd counts (3, 5, 7).

Kubernetes hub

A distributed key-value store that serves as Kubernetes' primary backing store for all cluster state and configuration data, making it the single source of truth for the entire control plane including nodes, pods, secrets, and configs.

etcd — Extended Technical Detail

What is etcd in Simple Terms?

etcd is the memory of your entire Kubernetes cluster. Every object you create — pods, deployments, secrets, configmaps, nodes — is stored in etcd. If etcd goes down or gets corrupted, your entire cluster loses its state. It is the most critical component to protect and back up.

Bash

+------------------------------------------+
| kubectl apply -f deployment.yaml         | <- You submit a resource
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| API Server validates the request         | <- Authorisation + validation
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| etcd stores the desired state            | <- Single source of truth
| "5 replicas of trading-api expected"     |    persisted to disk
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Controller Manager reads from etcd       | <- Reconciliation loop begins
| Scheduler reads from etcd               |    to match actual to desired
+------------------------------------------+

What etcd Stores

◈ DIAGRAM

+-----------------------------+
| Kubernetes Secrets          | <- Base64 encoded (encrypt at rest in prod)
+-----------------------------+
| ConfigMaps                  | <- App configuration data
+-----------------------------+
| Pod and Deployment specs    | <- All workload desired state
+-----------------------------+
| Node registrations          | <- kubelet heartbeats and node status
+-----------------------------+
| RBAC roles and bindings     | <- All access control policies
+-----------------------------+
| Service endpoints           | <- ClusterIP, port mappings
+-----------------------------+
| CRD objects                 | <- Custom resources (Argo, Istio, etc.)
+-----------------------------+

etcd Cluster Architecture — Why 3 or 5 Nodes

etcd uses the Raft consensus protocol. It requires a quorum (majority) of members to be available to accept writes:

◈ DIAGRAM

+------------------------+    +------------------------+    +------------------------+
| etcd-node-1 (leader)   |    | etcd-node-2 (follower) |    | etcd-node-3 (follower) |
|                        |    |                        |    |                        |
| mumbai-control-1       | -> | mumbai-control-2       | -> | mumbai-control-3       |
| Accepts writes         |    | Replicates from leader |    | Replicates from leader |
+------------------------+    +------------------------+    +------------------------+
 
3-node cluster:  quorum = 2.  Can survive 1 node failure.
5-node cluster:  quorum = 3.  Can survive 2 node failures.
Single node:     quorum = 1.  Zero fault tolerance — one outage = full cluster loss.

How to Check etcd Health

Bash

# Check etcd pod status in the control plane namespace
kubectl get pods -n kube-system | grep etcd
 
# Check etcd cluster health using etcdctl (run from a control plane node)
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health
 
# Expected healthy output:
# https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 3.2ms
 
# Check all members of the etcd cluster
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list
 
# Output:
# 3a57933972cb5131, started, mumbai-control-1, https://10.0.1.10:2380, https://10.0.1.10:2379
# 857c7a8d3a27c8b3, started, mumbai-control-2, https://10.0.1.11:2380, https://10.0.1.11:2379
# b6246cfd09d5e09c, started, mumbai-control-3, https://10.0.1.12:2380, https://10.0.1.12:2379

How to Back Up etcd

Bash

# Take a snapshot backup — automate this daily in production
ETCDCTL_API=3 etcdctl snapshot save \
  /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
 
# Verify the snapshot integrity immediately after saving
ETCDCTL_API=3 etcdctl snapshot status \
  /backup/etcd-snapshot-20250610-0200.db \
  --write-out=table
 
# Output:
# +----------+----------+------------+------------+
# |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
# +----------+----------+------------+------------+
# | fe01cf57 |   198008 |       1521 |     3.7 MB |
# +----------+----------+------------+------------+

How to Restore etcd from Backup

Bash

# Step 1 — Restore the snapshot to a new data directory
ETCDCTL_API=3 etcdctl snapshot restore \
  /backup/etcd-snapshot-20250610-0200.db \
  --data-dir=/var/lib/etcd-restored \
  --name=mumbai-control-1 \
  --initial-cluster="mumbai-control-1=https://10.0.1.10:2380" \
  --initial-advertise-peer-urls=https://10.0.1.10:2380
 
# Step 2 — Update the etcd static pod manifest to point to the new data dir
# Edit /etc/kubernetes/manifests/etcd.yaml:
# --data-dir=/var/lib/etcd-restored
 
# Step 3 — kubelet auto-restarts etcd with the restored state

etcd Encryption at Rest

By default, Kubernetes Secrets are stored in etcd as base64 — not encrypted. Enable encryption using an EncryptionConfiguration:

YAML

# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:              # AES-CBC encryption
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}        # Fallback: existing unencrypted secrets still readable

Bash

# Apply by adding this flag to kube-apiserver static pod manifest:
# --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
 
# Verify a secret is encrypted in etcd (should show gibberish, not base64 text)
ETCDCTL_API=3 etcdctl get \
  /registry/secrets/production/db-password \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C | head

Troubleshooting Common etcd Problems

Problem	Symptom	Fix
etcd pod CrashLoopBackOff	All kubectl commands fail with `etcdserver: no leader`	Check data directory permissions and disk space — etcd stops writing when disk is full
etcd leader election loop	Frequent leader changes in member list output	High network latency between control plane nodes — check inter-node latency, etcd needs <10ms
Slow API server responses	kubectl commands take 5-30 seconds	etcd has too many keys — run `etcdctl defrag` to compact and defragment the database
Snapshot restore failed	`snapshot file doesn't exist`	Wrong path or snapshot corrupted — always run `snapshot status` after saving to validate
Secrets visible as plain text in etcd	`etcdctl get` shows base64 text	Encryption at rest not enabled — add `EncryptionConfiguration` to kube-apiserver

Security
etcd stores Kubernetes Secrets in base64 encoding — not encrypted by default. Anyone with direct etcd access can read all secrets in the cluster. Enable encryption at rest using EncryptionConfiguration. For a platform like Zerodha handling financial transaction data and API keys, unencrypted etcd is a critical compliance gap.

Remember
Always run etcd as a 3-node or 5-node cluster in production for high availability. A single etcd node is a single point of failure for your entire cluster — one node crash means no new pods, no config changes, and no service deployments until it recovers.

Tip
Set up an automated daily etcd snapshot job via a CronJob that writes to an off-cluster storage bucket (AWS S3 ap-south-1, GCS). The snapshot is only useful if it is stored somewhere other than the cluster itself — a cluster-wide failure would destroy snapshots stored on the same nodes.

Common Mistake
Running etcdctl without the ETCDCTL_API=3 prefix. The default API version is v2, which uses a different command syntax and will silently return empty results or errors when your cluster is running etcd v3. Always prefix with ETCDCTL_API=3.

Frequently Asked Questions

Why does Kubernetes use etcd specifically instead of a general-purpose database for cluster state?

etcd is built on the Raft consensus algorithm, giving it strong consistency guarantees across a cluster of etcd nodes (typically 3 or 5) even during network partitions or node failures — critical because the Kubernetes API server relies on etcd's data being correct, not eventually consistent, when scheduling decisions and controller reconciliation depend on it. Its watch API also lets controllers efficiently react to state changes in real time rather than polling.

What's the most common operational mistake teams make with etcd in production clusters?

Not taking regular etcd snapshots (`etcdctl snapshot save`) and not testing restore procedures — etcd holds literally every piece of cluster state including Secrets, so losing quorum without a backup means rebuilding the entire cluster's configuration from scratch. Also common: running an even number of etcd members (e.g. 4) which doesn't improve fault tolerance over an odd number but does increase the chance of split votes; always run odd counts (3, 5, 7).