Kubelet

Kubernetes hub

The primary node agent in Kubernetes that runs on every worker node, responsible for ensuring containers described in PodSpecs are running, healthy, and reporting status back to the control plane.

Kubelet — Extended Technical Detail

What is the Kubelet in Simple Terms?

Think of the kubelet as the local manager on every worker node. The Kubernetes control plane (API server) sends orders — "run this pod with these containers." The kubelet on each node receives those orders and makes sure they are actually executed on that specific machine.

◈ DIAGRAM

+------------------------------------------+
| Kubernetes API Server                    | <- Control plane — issues PodSpecs
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet (on each worker node)            | <- Watches API, enforces PodSpecs locally
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Container Runtime (containerd)           | <- Actually starts/stops containers
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Running Pod                              | <- Final result: container is alive
+------------------------------------------+

What the Kubelet Does (Reconciliation Loop)

The kubelet runs a continuous reconciliation loop — comparing the desired state from the API server against the actual state on the node:

Step	Action
1. Watch	Polls the API server for PodSpecs assigned to its node
2. Pull	Pulls container images if not already cached on the node
3. Start	Starts containers via the container runtime (containerd)
4. Probe	Runs liveness and readiness probes at configured intervals
5. Report	Sends pod status updates back to the API server
6. Evict	Evicts pods when the node hits memory or disk pressure thresholds

How to Check Kubelet Status on a Node

Bash

# SSH into a worker node
ssh rahul@mumbai-prod-node-1
 
# Check if kubelet service is running
sudo systemctl status kubelet
 
# View live kubelet logs (last 10 minutes)
sudo journalctl -u kubelet -f --since "10 minutes ago"
 
# Check kubelet version and config
kubelet --version
sudo cat /var/lib/kubelet/config.yaml

Node NotReady — The Most Common Kubelet Issue

Bash

# Spot a NotReady node from the control plane
kubectl get nodes
# NAME                    STATUS     ROLES    AGE
# mumbai-prod-node-1      Ready      <none>   12d
# mumbai-prod-node-2      NotReady   <none>   12d   <- problem node
 
# Get details on why it is NotReady
kubectl describe node mumbai-prod-node-2
# Look for these conditions at the bottom:
# MemoryPressure    False
# DiskPressure      True   <- disk is full
# PIDPressure       False
# Ready             False  <- kubelet stopped reporting
 
# SSH into the node and restart kubelet
ssh rahul@10.0.1.51
sudo systemctl restart kubelet
sudo systemctl status kubelet

Kubelet Eviction — When Nodes Run Low on Resources

The kubelet automatically evicts pods when the node hits resource pressure thresholds. These defaults can be tuned in the kubelet config:

YAML

# /var/lib/kubelet/config.yaml — kubelet eviction thresholds
evictionHard:
  memory.available: "200Mi"      # Evict pods if less than 200Mi RAM free
  nodefs.available: "10%"        # Evict if disk drops below 10%
  nodefs.inodesFree: "5%"        # Evict if inodes drop below 5%
evictionSoft:
  memory.available: "500Mi"      # Warn first, then evict after grace period
evictionSoftGracePeriod:
  memory.available: "1m30s"      # Give pods 90 seconds before forced eviction

Kubelet Probes — How Pods Get Health-Checked

The kubelet is responsible for running three types of probes against containers:

◈ DIAGRAM

+---------------------------+     +---------------------------+     +---------------------------+
| Liveness Probe            |     | Readiness Probe           |     | Startup Probe             |
|                           |     |                           |     |                           |
| Is the app still alive?   |     | Is the app ready to recv  |     | Did the app start OK?     |
| Fail -> kubelet restarts  |     | traffic? Fail -> removed  |     | Replaces liveness during  |
| the container             |     | from Service endpoints    |     | slow init containers      |
+---------------------------+     +---------------------------+     +---------------------------+

YAML

# Example: all three probes on a Razorpay payment API container
containers:
  - name: payment-api
    image: registry.razorpay.in/payment-api:v2.4.1
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      failureThreshold: 30        # Allow up to 30 x 10s = 5 minutes to start
      periodSeconds: 10
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 15
      failureThreshold: 3         # Restart container after 3 consecutive failures
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 10
      failureThreshold: 2         # Remove from load balancer after 2 failures

Kubelet Troubleshooting Reference

Symptom	Likely Cause	Fix
Node shows `NotReady`	Kubelet crashed or lost API connectivity	`systemctl restart kubelet` on the node
`DiskPressure` on node	`/var/lib/kubelet` or `/var/log` full	Clear old images: `crictl rmi --prune`
`MemoryPressure` on node	Too many pods, no resource limits set	Add LimitRange to namespaces
Pods evicted unexpectedly	Kubelet hit eviction threshold	Check `kubectl describe node` for pressure conditions
`container runtime is down`	containerd service crashed	`systemctl restart containerd` then `systemctl restart kubelet`

Remember
If a node shows NotReady, the kubelet on that node has either crashed or lost connectivity to the API server. SSH into the node and run sudo systemctl status kubelet immediately — do not wait for the node to self-recover.

Security
The kubelet exposes port 10250 (read-write API). Unauthorized access to port 10250 gives full control over every pod on that node — including exec into running containers. Always firewall this port and restrict access to the API server and monitoring agents only. Port 10255 (read-only, deprecated) should be disabled entirely.

Tip
On Hotstar-scale clusters running 500+ nodes, kubelet log noise can be overwhelming. Use sudo journalctl -u kubelet --since "5 minutes ago" | grep -E "ERROR|WARN|evict" to filter signal from noise fast during an incident.

Common Mistake
Restarting a node to fix a NotReady status without first draining it. Always run kubectl drain mumbai-prod-node-2 --ignore-daemonsets --delete-emptydir-data before rebooting a node — otherwise all pods on it are hard-killed with no graceful termination.

Frequently Asked Questions

What exactly does the kubelet do that distinguishes it from the container runtime?

The kubelet doesn't run containers itself — it watches the API server for PodSpecs assigned to its node, then delegates actual container lifecycle operations (create, start, stop) to a container runtime (containerd, CRI-O) via the Container Runtime Interface (CRI). The kubelet's own job is reconciliation: continuously comparing desired pod state to observed state, running liveness/readiness probes, reporting node and pod status back to the control plane, and mounting volumes.

What happens to running pods on a node if the kubelet process crashes or the node loses contact with the control plane?

Pods already running keep running — the kubelet isn't in the data path for existing traffic, only for lifecycle management. But the node stops reporting heartbeats, and after the node's status goes NotReady for the pod-eviction-timeout (5 minutes by default), the control plane marks its pods for eviction and reschedules them elsewhere, even though the original pods may still technically be alive on the unreachable node — a scenario that can cause duplicate processing if you're not careful with idempotency.