DaemonSet

Kubernetes hub

A Kubernetes controller that ensures exactly one copy of a pod runs on every node in the cluster (or a filtered subset). When new nodes join, the DaemonSet automatically schedules a pod on them. When nodes leave, those pods are garbage collected. Used for infrastructure agents that must run everywhere — log collectors, monitoring exporters, network plugins, and security scanners.

DaemonSet — One Agent Per Node, Always

What is a DaemonSet in Simple Terms?

A DaemonSet is a standing order to the scheduler: "This pod must always run on every node — one copy per node, no more, no less." Not one total replica. One per node. When a new node joins the cluster, the DaemonSet pod is automatically placed on it. When a node is removed, the pod is cleaned up automatically.

DaemonSet vs Deployment — The Key Difference

◈ DIAGRAM

+------------------------------------------+     +------------------------------------------+
|             Deployment                   |     |              DaemonSet                   |
|                                          |     |                                          |
|  You control replica count               |     |  Cluster controls replica count          |
|  replicas: 3 (on any 3 nodes)            |     |  1 pod per node (on ALL nodes)           |
|                                          |     |                                          |
|  Use for: application workloads          |     |  Use for: infrastructure agents          |
+------------------------------------------+     +------------------------------------------+

One Pod Per Node — How It Looks

◈ DIAGRAM

+---------------+  +---------------+  +---------------+  +---------------+
|  mumbai-node-1|  |  mumbai-node-2|  |  mumbai-node-3|  |  mumbai-node-4|
|               |  |               |  |               |  |               |
|  [fluentd-0]  |  |  [fluentd-1]  |  |  [fluentd-2]  |  |  [fluentd-3]  |
+---------------+  +---------------+  +---------------+  +---------------+
        ^                                                          ^
        |                                                          |
        New node added to cluster -----> DaemonSet auto-schedules pod here

When to Use a DaemonSet

Use DaemonSet for infrastructure agents:

Log collection — Fluentd, Filebeat, Promtail (must read log files from every node's disk)
Metrics scraping — Node Exporter (must collect CPU/memory/disk from every node)
Network plugins — Calico, Cilium CNI agents (must configure networking on every node)
Security agents — Falco, CrowdStrike (must inspect every node's syscalls and processes)
Storage drivers — CSI node drivers that attach and mount volumes

Do NOT use DaemonSet for:

Application workloads (use Deployment)
Batch processing (use Jobs or CronJobs)
Anything where you want explicit control over replica count

A Real DaemonSet — Node Exporter for Prometheus

YAML

# node-exporter-daemonset.yaml
# Runs Prometheus Node Exporter on every node in mumbai-prod-cluster
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true              # Uses the node's network namespace directly
      hostPID: true                  # Sees all processes on the node (required for metrics)
      tolerations:
        - operator: Exists           # Tolerate ALL taints — run on control-plane nodes too
          effect: NoSchedule
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.7.0
          ports:
            - containerPort: 9100
              hostPort: 9100         # Binds directly to the node's port 9100
          args:
            - '--path.procfs=/host/proc'
            - '--path.sysfs=/host/sys'
          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: sys
              mountPath: /host/sys
              readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc              # Mounts the node's /proc filesystem
        - name: sys
          hostPath:
            path: /sys               # Mounts the node's /sys filesystem

Targeting Specific Nodes — Not Always Every Node

You do not always want a DaemonSet on every node. A GPU monitoring agent should only run on GPU nodes:

YAML

# nodeSelector — simple label match
spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-gpu      # Only schedule on nodes labelled as GPU nodes

For more complex targeting, use nodeAffinity:

YAML

# nodeAffinity — skip control-plane, run only on worker nodes
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node-role
              operator: In
              values:
                - worker             # Excludes control-plane nodes explicitly

Tolerations — Getting onto Tainted Nodes

Control-plane nodes carry a default taint: node-role.kubernetes.io/control-plane:NoSchedule. Without a matching toleration, your DaemonSet will skip them. The operator: Exists toleration bypasses ALL taints — use it for critical agents that must run everywhere like Falco or CNI plugins.

YAML

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule              # Specific — only bypass this one taint

Key DaemonSet Commands

Task	Command
List DaemonSet status	`kubectl get ds -n monitoring`
See which nodes have the pod	`kubectl get pods -o wide -n monitoring`
Check rollout status	`kubectl rollout status ds/node-exporter -n monitoring`
Force restart all pods	`kubectl rollout restart ds/node-exporter -n monitoring`
Describe DaemonSet events	`kubectl describe ds/node-exporter -n monitoring`
Check pod count vs node count	`kubectl get ds node-exporter -n monitoring`

Bash

# Verify a DaemonSet pod is running on every node
kubectl get pods -n monitoring -l app=node-exporter -o wide
 
# Output should show one pod per node:
# NAME                  READY   NODE
# node-exporter-4xk2p   1/1     mumbai-node-1
# node-exporter-7hq9r   1/1     mumbai-node-2
# node-exporter-m2pzn   1/1     mumbai-node-3

Security
Setting hostNetwork: true and hostPID: true gives the container full visibility into the host's network stack and every process running on the node. Only use these flags for trusted infrastructure agents like Node Exporter or Falco — never for application workloads. In Hotstar or PhonePe production clusters, DaemonSet pods with host access should be reviewed as part of every security audit.

Remember
DaemonSet pod count equals node count. If your cluster has 12 nodes and your DaemonSet shows 10 pods, two nodes have issues — either they are tainted without a matching toleration, or the pods are failing on those nodes. Use kubectl get pods -o wide to identify which nodes are missing coverage.

Common Mistake
Using a Deployment with replicas matching your node count as a substitute for a DaemonSet. If a node is added later, the Deployment will not automatically place a pod on it — you are back to manual scaling. Always use a DaemonSet for anything that must run on every node.

Tip
In Zerodha or Swiggy-scale clusters, DaemonSets for log collection (Fluentd/Promtail) are often the highest-volume pods in the cluster. Set proper resources.requests and resources.limits on them — an unthrottled Fluentd pod can consume enough CPU to starve application pods on the same node during log bursts.

Frequently Asked Questions

Why not just use a Deployment with enough replicas to cover every node?

A Deployment's replica count is a static number you have to update manually as nodes are added or removed, and the scheduler doesn't guarantee one pod per node — you could end up with two pods on one node and zero on another. A DaemonSet ties pod count directly to cluster topology: it watches the node list and schedules exactly one pod per matching node automatically, which is what you need for per-node agents like Fluentd or Node Exporter.

What's a common mistake teams make when running DaemonSets?

Not setting resource requests/limits on the DaemonSet pod. Since it runs on every node including ones running critical workloads, an unbounded log collector or scanner can starve application pods for CPU or memory during a spike. Also worth watching: DaemonSet pods bypass normal scheduler resource-based placement by default (they tolerate most taints), so they'll land on nodes you might have wanted kept clear, like dedicated GPU nodes, unless you add explicit tolerations/node selectors.