Troubleshooting Kubernetes Pod OOMKilled and CrashLoopBackOff Errors

Production Best Practices & Common Pitfalls

• Always test cross-namespace connectivity after adding any NetworkPolicy. A policy in the target namespace affects all inbound traffic including from other namespaces that previously worked without any policy.
• Use Cilium with Hubble in production — the real-time policy trace and drop visibility is worth the migration cost. Debugging NetworkPolicies without Hubble is guesswork.
• Label your namespaces explicitly with kubernetes.io/metadata.name — this label is auto-applied in Kubernetes 1.21+ and is required for reliable namespace-based NetworkPolicy selectors.
• Monitor CoreDNS with Prometheus and alert on coredns_dns_response_rcode_count_total{rcode="SERVFAIL"} — a spike indicates upstream DNS failure that will cause cascading service discovery failures across the entire cluster.
• Never run tcpdump directly on a node in production without approval — packet capture on a financial services cluster is a compliance event that must be logged and justified.

🔴 Common Mistake: Checking pod logs to diagnose networking issues. Application logs say "connection refused" or "timeout" — they cannot tell you whether the failure is at Layer 2 (CNI), Layer 3 (kube-proxy), Layer 4 (DNS), or Layer 5 (NetworkPolicy). Always use network-level tools like netshoot, not application logs, for network debugging.

Quick Reference & Troubleshooting Commands