Debugging Kubernetes Networking with kubectl and CNI Plugins

Debugging Kubernetes Networking with kubectl and CNI Plugins

Overview and What You Will Learn

Kubernetes networking failures are among the hardest production issues to diagnose — a pod can be running perfectly but completely unable to reach another service due to a CNI misconfiguration, a missing NetworkPolicy rule, a DNS resolution failure, or a kube-proxy iptables issue. This lab walks you through a systematic debugging methodology for every layer of Kubernetes networking using real kubectl commands, the netshoot debug container, and CNI-specific diagnostic tools.

By the end of this guide you will be able to:

• Diagnose pod-to-pod, pod-to-service, and pod-to-external connectivity failures systematically
• Debug DNS resolution failures using CoreDNS logs and nslookup inside pods
• Identify CNI plugin misconfiguration causing pods to stay in ContainerCreating state
• Trace kube-proxy iptables rules to verify Service routing is correctly programmed
• Write and debug Kubernetes NetworkPolicies that restrict inter-pod traffic

Why This Matters in Production

At Zerodha, a silent network partition between the order matching engine and the risk management service caused trades to execute without risk checks for 4 minutes during a CNI upgrade — not detected by application logs because the service call was timing out silently and falling back to a cached value. The incident was only discovered through a networking-level trace.

Kubernetes networking has five distinct layers that can fail independently — the pod network interface, CNI plugin, kube-proxy service rules, CoreDNS, and NetworkPolicies. Engineers who cannot debug each layer independently will waste hours on production incidents that a systematic approach resolves in minutes.

Core Principles

The five networking layers in Kubernetes and their failure modes: Layer 1 — Pod Network Interface (veth pair + CNI assignment) Failure: Pod stuck in ContainerCreating, no IP assigned Tool: kubectl describe pod, ip addr inside pod Layer 2 — CNI Plugin (Calico / Cilium / Flannel overlay) Failure: Pod has IP but cannot reach pods on OTHER nodes Tool: CNI logs, calicoctl, cilium status Layer 3 — kube-proxy (iptables / IPVS Service rules) Failure: Pod can reach pod IPs directly but Service IP fails Tool: iptables -L -n, kubectl get endpoints Layer 4 — CoreDNS (cluster DNS resolution) Failure: Service IPs work but DNS names fail Tool: kubectl logs coredns, nslookup inside pod Layer 5 — NetworkPolicy (ingress/egress firewall rules) Failure: Connection refused or timeout with no obvious cause Tool: kubectl get networkpolicies, policy trace tools

Always debug from Layer 1 upward — never assume the problem is DNS when the pod might not have a network interface at all.

Detailed Step-by-Step Practical Lab

Step 1 — Deploy the netshoot Debug Container

netshoot is the standard Swiss Army knife for Kubernetes network debugging — it contains curl, nslookup, dig, tcpdump, netstat, traceroute, and dozens of other tools:

📌 Remember: kubectl debug --target=<pod> shares the pod's network namespace — it sees the same IP, the same routes, and the same DNS config as the application pod. This is the most accurate way to reproduce networking issues exactly as the application experiences them.

Step 2 — Debug Layer 1: Verify Pod Has a Network Interface and IP

Step 3 — Debug Layer 2: Test Pod-to-Pod Connectivity Across Nodes

Step 4 — Debug Layer 3: Verify kube-proxy Service Rules

💡 Tip: If the Service has correct endpoints but the ClusterIP still fails, the kube-proxy iptables rules may be stale. Restarting the kube-proxy DaemonSet pod on the affected node forces a full iptables resync: kubectl delete pod kube-proxy-xxxxx -n kube-system.

Step 5 — Debug Layer 4: CoreDNS and DNS Resolution Failures

Step 6 — Debug Layer 5: NetworkPolicy Blocking Traffic

⚠️ Security: Once you create any NetworkPolicy in a namespace, all traffic not explicitly allowed is denied by default. This means adding your first NetworkPolicy to a namespace can instantly break all existing connectivity if you don't also add allow rules for every legitimate traffic flow.

Step 7 — Run a Full Connectivity Matrix Test

For a comprehensive network health check across all services:

Production Best Practices & Common Pitfalls

• Always test cross-namespace connectivity after adding any NetworkPolicy. A policy in the target namespace affects all inbound traffic including from other namespaces that previously worked without any policy.
• Use Cilium with Hubble in production — the real-time policy trace and drop visibility is worth the migration cost. Debugging NetworkPolicies without Hubble is guesswork.
• Label your namespaces explicitly with kubernetes.io/metadata.name — this label is auto-applied in Kubernetes 1.21+ and is required for reliable namespace-based NetworkPolicy selectors.
• Monitor CoreDNS with Prometheus and alert on coredns_dns_response_rcode_count_total{rcode="SERVFAIL"} — a spike indicates upstream DNS failure that will cause cascading service discovery failures across the entire cluster.
• Never run tcpdump directly on a node in production without approval — packet capture on a financial services cluster is a compliance event that must be logged and justified.

🔴 Common Mistake: Checking pod logs to diagnose networking issues. Application logs say "connection refused" or "timeout" — they cannot tell you whether the failure is at Layer 2 (CNI), Layer 3 (kube-proxy), Layer 4 (DNS), or Layer 5 (NetworkPolicy). Always use network-level tools like netshoot, not application logs, for network debugging.

Quick Reference & Troubleshooting Commands

Asset Tracker Update: