What is Kubelet? | DevOps Dictionary

Kubelet — Extended Technical Detail

What is the Kubelet in Simple Terms?

Think of the kubelet as the local manager on every worker node. The Kubernetes control plane (API server) sends orders — "run this pod with these containers." The kubelet on each node receives those orders and makes sure they are actually executed on that specific machine.

◈ DIAGRAM

+------------------------------------------+
| Kubernetes API Server                    | <- Control plane — issues PodSpecs
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet (on each worker node)            | <- Watches API, enforces PodSpecs locally
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Container Runtime (containerd)           | <- Actually starts/stops containers
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Running Pod                              | <- Final result: container is alive
+------------------------------------------+

What the Kubelet Does (Reconciliation Loop)

The kubelet runs a continuous reconciliation loop — comparing the desired state from the API server against the actual state on the node:

Step	Action
1. Watch	Polls the API server for PodSpecs assigned to its node
2. Pull	Pulls container images if not already cached on the node
3. Start	Starts containers via the container runtime (containerd)
4. Probe	Runs liveness and readiness probes at configured intervals
5. Report	Sends pod status updates back to the API server
6. Evict	Evicts pods when the node hits memory or disk pressure thresholds

How to Check Kubelet Status on a Node

Bash

1# SSH into a worker node
2ssh rahul@mumbai-prod-node-1
3 
4# Check if kubelet service is running
5sudo systemctl status kubelet
6 
7# View live kubelet logs (last 10 minutes)
8sudo journalctl -u kubelet -f --since "10 minutes ago"
9 
10# Check kubelet version and config
11kubelet --version
12sudo cat /var/lib/kubelet/config.yaml

Node NotReady — The Most Common Kubelet Issue

Bash

1# Spot a NotReady node from the control plane
2kubectl get nodes
3# NAME                    STATUS     ROLES    AGE
4# mumbai-prod-node-1      Ready      <none>   12d
5# mumbai-prod-node-2      NotReady   <none>   12d   <- problem node
6 
7# Get details on why it is NotReady
8kubectl describe node mumbai-prod-node-2
9# Look for these conditions at the bottom:
10# MemoryPressure    False
11# DiskPressure      True   <- disk is full
12# PIDPressure       False
13# Ready             False  <- kubelet stopped reporting
14 
15# SSH into the node and restart kubelet
16ssh rahul@10.0.1.51
17sudo systemctl restart kubelet
18sudo systemctl status kubelet

Kubelet Eviction — When Nodes Run Low on Resources

The kubelet automatically evicts pods when the node hits resource pressure thresholds. These defaults can be tuned in the kubelet config:

YAML

1# /var/lib/kubelet/config.yaml — kubelet eviction thresholds
2evictionHard:
3  memory.available: "200Mi"      # Evict pods if less than 200Mi RAM free
4  nodefs.available: "10%"        # Evict if disk drops below 10%
5  nodefs.inodesFree: "5%"        # Evict if inodes drop below 5%
6evictionSoft:
7  memory.available: "500Mi"      # Warn first, then evict after grace period
8evictionSoftGracePeriod:
9  memory.available: "1m30s"      # Give pods 90 seconds before forced eviction

Kubelet Probes — How Pods Get Health-Checked

The kubelet is responsible for running three types of probes against containers:

◈ DIAGRAM

+---------------------------+     +---------------------------+     +---------------------------+
| Liveness Probe            |     | Readiness Probe           |     | Startup Probe             |
|                           |     |                           |     |                           |
| Is the app still alive?   |     | Is the app ready to recv  |     | Did the app start OK?     |
| Fail -> kubelet restarts  |     | traffic? Fail -> removed  |     | Replaces liveness during  |
| the container             |     | from Service endpoints    |     | slow init containers      |
+---------------------------+     +---------------------------+     +---------------------------+

YAML

1# Example: all three probes on a Razorpay payment API container
2containers:
3  - name: payment-api
4    image: registry.razorpay.in/payment-api:v2.4.1
5    startupProbe:
6      httpGet:
7        path: /healthz
8        port: 8080
9      failureThreshold: 30        # Allow up to 30 x 10s = 5 minutes to start
10      periodSeconds: 10
11    livenessProbe:
12      httpGet:
13        path: /healthz
14        port: 8080
15      initialDelaySeconds: 5
16      periodSeconds: 15
17      failureThreshold: 3         # Restart container after 3 consecutive failures
18    readinessProbe:
19      httpGet:
20        path: /ready
21        port: 8080
22      periodSeconds: 10
23      failureThreshold: 2         # Remove from load balancer after 2 failures

Kubelet Troubleshooting Reference

Symptom	Likely Cause	Fix
Node shows `NotReady`	Kubelet crashed or lost API connectivity	`systemctl restart kubelet` on the node
`DiskPressure` on node	`/var/lib/kubelet` or `/var/log` full	Clear old images: `crictl rmi --prune`
`MemoryPressure` on node	Too many pods, no resource limits set	Add LimitRange to namespaces
Pods evicted unexpectedly	Kubelet hit eviction threshold	Check `kubectl describe node` for pressure conditions
`container runtime is down`	containerd service crashed	`systemctl restart containerd` then `systemctl restart kubelet`

📌 Remember: If a node shows NotReady, the kubelet on that node has either crashed or lost connectivity to the API server. SSH into the node and run sudo systemctl status kubelet immediately — do not wait for the node to self-recover.

⚠️ Security: The kubelet exposes port 10250 (read-write API). Unauthorized access to port 10250 gives full control over every pod on that node — including exec into running containers. Always firewall this port and restrict access to the API server and monitoring agents only. Port 10255 (read-only, deprecated) should be disabled entirely.

💡 Tip: On Hotstar-scale clusters running 500+ nodes, kubelet log noise can be overwhelming. Use sudo journalctl -u kubelet --since "5 minutes ago" | grep -E "ERROR|WARN|evict" to filter signal from noise fast during an incident.

🔴 Common Mistake: Restarting a node to fix a NotReady status without first draining it. Always run kubectl drain mumbai-prod-node-2 --ignore-daemonsets --delete-emptydir-data before rebooting a node — otherwise all pods on it are hard-killed with no graceful termination.

Syncing Data

Kubelet

Technical Explanation & Usage