Kubernetes Interview Questions

A Pod is the smallest deployable unit in Kubernetes: one or more containers that share the same network namespace (one IP address, one port space) and can share storage volumes. Containers in the same Pod talk to each other over localhost.

Kubernetes schedules Pods rather than bare containers because some workloads genuinely need more than one container acting as a single unit — a main application container plus a sidecar that ships logs, proxies traffic, or syncs files. Putting them in one Pod guarantees they land on the same node, start and stop together, and can share a filesystem without any extra networking setup. Most Pods in practice still run a single container, but the Pod is the abstraction Kubernetes reasons about for scheduling, networking, and lifecycle — never the container directly.

The common trap: describing a Pod as "just a wrapper around one container." That's true in the majority of real deployments, but it misses why the abstraction exists at all, which is usually the actual follow-up question.

Reference: Kubernetes docs — Pods

A ReplicaSet's only job is to keep a specified number of identical Pods running — if one dies, it starts another. A Deployment sits one layer above the ReplicaSet and manages it for you, adding rolling updates, rollbacks, and revision history.

In practice you almost never create a ReplicaSet directly. When you change a Deployment's Pod template — a new image tag, an updated environment variable — the Deployment controller creates a new ReplicaSet and gradually shifts Pods from the old one to the new one, rather than editing Pods in place. The old ReplicaSet is kept around (scaled to zero) so kubectl rollout undo has something to roll back to. Scaling a Deployment up or down doesn't create a new ReplicaSet; only a change to the Pod template does.

Reference: Kubernetes docs — Deployments

A Service gives a stable IP address and DNS name to a group of Pods that comes and goes as Pods are replaced. Without it, you'd have to track individual Pod IPs yourself, which is useless in practice because Pods are ephemeral — every restart, reschedule, or rolling update gives a Pod a new IP.

A Service selects Pods by label and continuously updates its list of healthy endpoints as Pods start, stop, and pass or fail readiness checks. Other Pods (or external clients, depending on the Service type) then talk to the Service's stable address instead of any individual Pod, and traffic gets load-balanced across whichever Pods are currently ready. This decoupling — "talk to the Service, not the Pod" — is the basic building block that makes rolling updates and self-healing possible without breaking anything that depends on the app.

Reference: Kubernetes docs — Service

Both store configuration data outside your container image and inject it into Pods as environment variables or mounted files — the difference is intent, not security. A ConfigMap is for non-sensitive values like feature flags, log levels, or a service's hostname. A Secret is for sensitive values like passwords, API keys, and TLS certificates.

The trap worth naming here: Secrets are base64-encoded, not encrypted, by default. Base64 is a reversible encoding, not a cipher — anyone with read access to the Secret object via the API, or read access to etcd directly, can decode it in one command. To actually protect Secret contents you need encryption at rest configured on the cluster and RBAC locked down so only the Pods and people that need a Secret can read it. Treat "it's a Secret object" as organizing metadata, not as a security control on its own.

Reference: Kubernetes docs — Secrets

A Namespace is a way to divide a single cluster into multiple virtual clusters, giving you a scope for names, RBAC rules, resource quotas, and NetworkPolicies. Two Deployments named api can coexist fine as long as they're in different Namespaces.

Reach for Namespaces when you need to separate environments (staging vs production on shared infrastructure), separate teams or tenants on a shared cluster, or want to apply different RBAC and quota rules to different parts of an application. They're not a hard security boundary by default — a Pod in one Namespace can usually still reach a Service in another unless a NetworkPolicy says otherwise — so Namespaces alone aren't enough for hard multi-tenant isolation between untrusted teams.

Reference: Kubernetes docs — Namespaces

A DaemonSet ensures that exactly one copy of a Pod runs on every node in the cluster (or every node matching a selector), and automatically adds a Pod when a new node joins.

A Deployment doesn't care which node a Pod lands on — it just wants N replicas running somewhere. A DaemonSet cares specifically about node coverage: you use it for per-node infrastructure like a log-shipping agent, a metrics collector, or a CNI network plugin, where you need "one instance everywhere," not "N instances wherever there's room." If you scaled a Deployment to match your node count instead, the scheduler could still pile two Pods on one node and zero on another — a DaemonSet is the tool that guarantees the one-per-node property directly.

Reference: Kubernetes docs — DaemonSet

etcd is the distributed key-value store that holds the entire state of a Kubernetes cluster — every object you can kubectl get, from Pods and Deployments to Secrets and RBAC rules, lives there. The API server is the only component that talks to etcd directly; everything else (scheduler, controllers, kubelets) reads and writes cluster state through the API server.

Because etcd holds everything, including Secrets, it's a high-value target: anyone with direct etcd access can read Secret values regardless of RBAC, unless encryption at rest is enabled. etcd also runs as a Raft consensus cluster (typically 3 or 5 members) for high availability, so losing quorum — more than half the members — makes the cluster read-only or unavailable even if every node is otherwise healthy. Regular etcd backups are considered non-negotiable in production for exactly this reason.

Reference: Kubernetes docs — etcd

kubectl create is imperative: it creates a resource from a manifest and fails if that resource already exists. kubectl apply is declarative: it creates the resource if it's missing, or computes a diff against the live object and patches just the fields that changed if it already exists.

This matters in practice because apply is what you put in CI/CD pipelines and GitOps workflows — you can run the same command repeatedly against version-controlled YAML and it converges the cluster to match the file, rather than erroring out on the second run. apply also keeps a record of the last-applied configuration as an annotation, which is how it knows what changed versus what a person or another controller modified out-of-band. create is mostly useful for quick one-off testing.

Reference: Kubernetes docs — Managing Resources

kube-proxy is the component that runs on every node and turns a Service's stable virtual IP into real traffic reaching one of the Service's backend Pods. A Service's IP isn't attached to any network card anywhere — it only becomes real because kube-proxy watches Services and their endpoints, then programs the node's networking layer with rules that intercept traffic to that IP and forward it to a Pod.

It does this without touching the application at all — a Pod calling a Service's ClusterIP has no idea kube-proxy exists, the redirection happens transparently at the OS networking layer. Historically it did this with iptables rules, and large clusters with many Services later adopted an IPVS mode for better performance at scale; some CNI plugins like Cilium now replace kube-proxy's job entirely with eBPF instead. Either way, the concept an interviewer wants to hear is: Services are a Kubernetes-level abstraction, and kube-proxy is the thing that makes that abstraction real on the wire.

Reference: Kubernetes docs — kube-proxy

A CRD lets you extend the Kubernetes API with your own object types, so kubectl get certificates or kubectl get databases works exactly like kubectl get pods — same API conventions, same kubectl verbs, same RBAC model — even though "Certificate" or "Database" isn't a built-in Kubernetes concept.

On its own, a CRD is just a schema: it tells the API server a new object type exists and what fields it has, and the API server will happily store and serve objects of that type. Nothing acts on those objects unless you also run a controller (often called an operator) that watches for them and does something in response — provisions a real database, requests a TLS certificate from a CA, configures a message queue. The CRD plus the controller together is what people usually mean by "an operator": the CRD defines the desired-state object, the controller reconciles the cluster toward it, the same pattern Kubernetes itself uses internally for Deployments and ReplicaSets.

Reference: Kubernetes docs — Custom Resources

An image is a static, read-only bundle of files — application code, dependencies, a filesystem layout — built once and stored in a registry. A container is a running instance of that image: an image plus a writable layer on top, plus the process(es), namespaces, and cgroup limits the container runtime gives it once it starts.

The relationship is the same as a class and an object in programming, or a recipe and a cooked meal: one image can be instantiated as many containers simultaneously, each with its own writable layer and its own lifecycle, and none of them affect the underlying image. This is why restarting a container throws away anything it wrote to its own filesystem (unless that path is backed by a volume) — the writable layer is thrown away and a fresh one is created from the same unchanged image. It's also why "the image is immutable" is a security property people rely on: you can't fix a running container by editing files inside it in any way that survives a restart; you rebuild the image and roll out a new version instead.

Reference: Kubernetes docs — Images

A sidecar is a second container in the same Pod as the main application container, running alongside it to provide supporting functionality — without being the thing that does the application's actual job. Because containers in a Pod share the same network namespace and can share volumes, a sidecar can transparently intercept traffic, tail log files, or sync data without the main application needing to know it's there.

Common real examples: a service mesh proxy (like Envoy in Istio) that every outbound and inbound request passes through for traffic policy and mTLS; a log-shipping agent that tails a shared volume and forwards entries to a logging backend; a file-sync container that pulls config or secrets from an external source on an interval. The defining trait of a "classic" sidecar is that it runs for the Pod's whole lifetime, not just at startup — which is what distinguishes it from an init container, and it's also why ordering sidecar startup and shutdown correctly used to be awkward before native sidecar support existed.

Reference: Kubernetes docs — Sidecar Containers

They're three levels of exposure, each building on the one before it.

A NodePort Service actually opens that port on every node, whether or not a Pod is running there, and routes the traffic on to a Pod via kube-proxy — this surprises people who expect the port to only be open on nodes running the workload. A LoadBalancer Service is built on top of a NodePort under the hood; the cloud provider's controller provisions an external load balancer that forwards to the NodePort. On bare-metal clusters with no cloud integration, LoadBalancer Services stay stuck in <pending> unless something like MetalLB is installed to fulfil the request.

Reference: Kubernetes docs — Service

Use an Ingress when you need to route HTTP/HTTPS traffic for many services through one entry point, based on hostname or URL path, with TLS termination — rather than giving every service its own cloud load balancer.

A LoadBalancer Service is one external IP per Service, which gets expensive and unwieldy once you have more than a handful of services, and it operates at layer 4 (it doesn't know about HTTP paths or hostnames at all). An Ingress resource describes routing rules — api.example.com goes here, example.com/blog goes there — and an Ingress controller (NGINX, Traefik, HAProxy, or a cloud-managed one) reads those rules and does the actual routing, usually sitting behind a single LoadBalancer Service. The trade-off is you now depend on the Ingress controller being installed and healthy; a LoadBalancer Service has no such dependency. For non-HTTP traffic (raw TCP/UDP, gRPC without HTTP/2 fronting, databases), Ingress isn't the right tool and you're back to LoadBalancer or NodePort. The newer Gateway API is the CNCF's designated successor to Ingress and is increasingly the answer in fresh clusters, but Ingress is still what most existing production clusters run.

Reference: Kubernetes docs — Ingress

A liveness probe answers "should this container be restarted?" A readiness probe answers "should this Pod currently receive traffic?" — and they cause different, easily confused, actions when they fail.

When a liveness probe fails, the kubelet kills and restarts the container, on the assumption that it's deadlocked or otherwise stuck in a bad state a restart can fix. When a readiness probe fails, Kubernetes just removes the Pod from the Service's list of endpoints — the container keeps running untouched, it simply stops receiving new traffic until the probe passes again. A startup probe exists to handle slow-starting apps: while it's still failing, the liveness and readiness probes are disabled entirely, so a genuinely slow boot doesn't get killed by an impatient liveness probe before it's had a chance to start.

The classic trap: a liveness probe that checks a downstream dependency (a database connection, for example). If that dependency has an outage, every Pod fails its liveness probe simultaneously and gets restarted in a loop — restarting the app does nothing to fix a database outage, so you've turned one outage into a self-inflicted crash loop on top of it. Liveness probes should be cheap, local checks; deeper dependency checks belong on the readiness probe instead.

Reference: Kubernetes docs — Configure Liveness, Readiness and Startup Probes

Nothing happens to the running Pods. A rolling update is only triggered by a change to .spec.template — the Pod template — not by any other field in the Deployment, and not just by running apply itself.

If you edit replicas, add an annotation outside the Pod template, or simply re-apply an identical file, the Deployment controller reconciles the object but doesn't create a new ReplicaSet or touch existing Pods, because the thing it actually watches for change is the template's content hash. This is why kubectl scale doesn't create a new revision in kubectl rollout history, and why you sometimes need kubectl rollout restart — which works by patching an annotation inside the Pod template specifically, so it does count as a template change and triggers a real rolling restart, useful when you want to pick up an updated ConfigMap or refresh Pods without changing the image.

Reference: Kubernetes docs — Performing a Rolling Update

A request is what the scheduler reserves for a container when deciding which node it fits on. A limit is the hard ceiling the container is never allowed to exceed once it's running. They solve two different problems — placement versus enforcement — and mixing them up is one of the most common ways real clusters get into trouble.

If requests are set too low relative to actual usage, the scheduler happily overcommits a node, and under real load every Pod on it starts competing for resources the node doesn't actually have. If limits are set too low, the container gets throttled or killed even though the node has capacity to spare. Requests and limits don't have to match: when they differ, the Pod gets a Burstable QoS class, meaning it can use more than its request when the node has spare capacity, up to its limit. When requests equal limits exactly, the Pod gets Guaranteed QoS, the highest eviction priority under node pressure.

Reference: Kubernetes docs — Resource Management for Pods and Containers

Exceeding a memory limit gets the container killed immediately (OOMKilled, exit code 137); exceeding a CPU limit just throttles it. Memory and CPU aren't handled symmetrically, and that asymmetry trips up a lot of debugging sessions.

Memory can't be reclaimed from a process without killing something — there's no "give some back" — so the kernel's OOM killer terminates the container the instant it crosses its cgroup memory limit, with no grace period and no warning in the application's own logs. CPU is fundamentally different: it's a rate, not a fixed quantity, so a container that hits its CPU limit is simply throttled — it gets scheduled less often, requests get slower — but it keeps running. This is why "the app looks fine in logs but requests are timing out" is very often a CPU limit set too tight, while "the app just vanishes and restarts" is very often OOMKilled. kubectl describe pod shows Reason: OOMKilled and Exit Code: 137 in the last-state section when it's a memory kill.

Reference: Kubernetes docs — Resource Management for Pods and Containers

Choose a StatefulSet when Pods need a stable, predictable identity and their own persistent storage that survives a reschedule — think databases, message queue brokers, or anything where "which specific instance am I talking to" matters. A Deployment assumes every Pod is interchangeable; a StatefulSet assumes they're not.

A StatefulSet gives each Pod a stable name and DNS entry (db-0, db-1, db-2, ...) that doesn't change even if the Pod is rescheduled to a different node, and pairs each ordinal with its own PersistentVolumeClaim, so db-1 always comes back with the same disk it had before. It also creates and scales Pods in strict order by default — db-0 must be Running and Ready before db-1 is created — which matters for things like a primary needing to exist before replicas join it. The trade-off is operational complexity: StatefulSets are slower to scale, harder to reason about during a rolling update, and deleting one doesn't delete its PersistentVolumeClaims by default, which is a safety feature but also a common source of "why is old data still there" surprises.

Reference: Kubernetes docs — StatefulSets

All three influence where Pods land, but they answer different questions and sit on different sides of the relationship. nodeSelector and node affinity are the Pod saying "I want a node like this." Taints and tolerations are the node saying "don't schedule here unless you're explicitly allowed."

nodeSelector is the simplest form: an exact-match label requirement, all-or-nothing. Node affinity does the same job with much more expressiveness — requiredDuringSchedulingIgnoredDuringExecution for hard constraints, preferredDuringSchedulingIgnoredDuringExecution for soft preferences the scheduler will try but won't block on, plus operators like In, NotIn, and Exists. Taints and tolerations work in the opposite direction: a taint on a node repels every Pod that doesn't carry a matching toleration, which is how you reserve nodes for a specific purpose — GPU nodes, a dedicated team's workloads, nodes mid-drain — without having to add affinity rules to every other Pod in the cluster to keep them away. In practice, dedicating a node pool usually means taint the nodes and add affinity on the Pods that should go there — the taint keeps everyone else off, the affinity actually steers the right Pods on.

Reference: Kubernetes docs — Assigning Pods to Nodes

The HPA runs a control loop that periodically compares observed metrics — usually CPU or memory utilization, optionally custom or external metrics — against a target you configured, and adjusts the replica count of a Deployment or StatefulSet to close the gap, within a min/max bound you set.

For CPU- or memory-based autoscaling it needs the Metrics Server installed in the cluster, since the HPA itself doesn't collect metrics, it only reads them from the metrics API. Scaling decisions aren't instant on every observation — the HPA has a stabilization window and won't scale down aggressively right after scaling up, specifically to avoid reacting to a single noisy spike. It's also worth knowing the HPA scales replica count, not per-Pod resources — for that you'd want the Vertical Pod Autoscaler instead, and running both against the same metric on the same workload at once is a known source of conflicting decisions.

Reference: Kubernetes docs — Horizontal Pod Autoscaling

A Role grants permissions within a single Namespace; a ClusterRole grants permissions across the whole cluster, or can be reused across multiple Namespaces via separate RoleBindings. Both list the same kind of thing — allowed verbs (get, list, create, delete, ...) on specific resource types — the difference is scope, not capability.

A Role bound with a RoleBinding only ever applies inside the Namespace it lives in. A ClusterRole is more flexible: bound with a ClusterRoleBinding it applies everywhere, but bound with a RoleBinding instead, it applies only within that RoleBinding's Namespace — a common pattern for reusing one "view-only" ClusterRole across many teams' namespaces without redefining the same rules repeatedly. RBAC permissions are purely additive with no explicit deny — a subject's effective permissions are the union of everything every applicable binding grants, which is why least-privilege design means starting from nothing and adding narrowly, not starting from cluster-admin and trying to subtract.

Reference: Kubernetes docs — Using RBAC Authorization

It depends entirely on how the ConfigMap got into the Pod, and this is one of the more common "worked in my head, not in prod" gaps. If it's mounted as a volume, the kubelet eventually syncs the updated file into the container — typically within about a minute — without restarting anything. If it's injected as an environment variable, it is fixed at Pod start and will never update on a running Pod; you have to recreate the Pod to pick up the new value.

Even for the volume-mount case, "the file updated" doesn't mean "the app noticed." Most applications read config once at startup and never watch the file for changes, so unless the app specifically implements a file watcher (or you're relying on a sidecar or kubectl rollout restart to force new Pods), the new file sits on disk unused until the next restart. This is exactly why kubectl rollout restart deployment/myapp is the practical way to force a ConfigMap change to actually take effect — it triggers a real rolling update rather than hoping every container polls its config file.

Reference: Kubernetes docs — ConfigMaps

A PersistentVolume (PV) is a piece of actual storage in the cluster — an EBS volume, an NFS share, a local disk — provisioned by an admin or dynamically by a StorageClass. A PersistentVolumeClaim (PVC) is a request for storage made by a Pod, specifying how much space and what access mode it needs. Kubernetes binds a PVC to a matching PV; the Pod only ever references the PVC, never the PV directly.

This indirection is the point: it decouples "what storage exists" from "what an application asked for," the same way a Service decouples Pod IPs from a stable address. With dynamic provisioning, a StorageClass creates the PV on demand when a PVC requests it, which is how most clusters actually work today — you rarely hand-create PVs. Access modes matter too: ReadWriteOnce (one node at a time), ReadOnlyMany, and ReadWriteMany (multiple nodes simultaneously, and only supported by some storage backends like NFS or EFS) — trying to attach an RWO volume to Pods on two different nodes at once is a common source of Pods stuck Pending with a mount error.

Reference: Kubernetes docs — Persistent Volumes

A PodDisruptionBudget (PDB) sets a floor on how many Pods of an application must stay available during a voluntary disruption — a node drain for maintenance, a cluster autoscaler scaling down, or someone running kubectl drain. Kubernetes will respect that floor and refuse to evict Pods that would breach it, until enough replacements come back up elsewhere.

The key word is voluntary: a PDB has no power over involuntary disruptions like a node crashing or a Pod being OOMKilled — it can't prevent those. What it prevents is an operator (human or the cluster autoscaler) unintentionally taking down every replica of a service at once during routine maintenance. You define it as either a minimum available count or a maximum unavailable count; for a 3-replica service you might require at least 2 available at all times, which stops a node drain from evicting two Pods simultaneously and leaving you with one.

Reference: Kubernetes docs — Specifying a Disruption Budget for your Application

A Job runs one or more Pods to completion, once, and tracks success rather than keeping Pods alive indefinitely the way a Deployment does. A CronJob is a Job that runs on a schedule, defined with standard cron syntax, creating a new Job (and its Pods) at each scheduled time.

A Job is what you use for a one-off task — a database migration, a batch export — and it retries failed Pods up to a configurable backoffLimit before giving up, unlike a Deployment which keeps restarting forever. A CronJob adds scheduling on top and has its own gotchas: concurrencyPolicy controls whether a new run is allowed to start while the previous one is still going (Allow, Forbid, or Replace), and startingDeadlineSeconds controls how late a missed run is still allowed to fire — both matter a lot for jobs where overlapping runs would corrupt data, or where a controller-manager outage causing a backlog of missed runs shouldn't all fire at once when it recovers.

Reference: Kubernetes docs — CronJob

An init container runs to completion before any of a Pod's regular containers start, in order if there are several, and the Pod won't proceed until every init container exits successfully. Regular containers in a Pod all start together and are expected to keep running; init containers are expected to finish and stop.

This ordering guarantee is the whole value: use an init container for setup that must happen before the app starts and that the app itself shouldn't be responsible for — waiting for a dependency to become reachable, running a one-time database migration, cloning a config repo into a shared volume, or fixing file permissions on a mounted volume. If an init container fails, the kubelet restarts it according to the Pod's restartPolicy, and the whole Pod stays stuck (visible as Init:CrashLoopBackOff or Init:Error) until it succeeds — which is why a Pod that never gets past Init needs kubectl logs <pod> -c <init-container-name>, not the main container's logs, to diagnose.

Reference: Kubernetes docs — Init Containers

A NetworkPolicy restricts which Pods can talk to which other Pods (and on which ports), enforced by the cluster's CNI plugin. Without any NetworkPolicy in a Namespace, the default is fully open: any Pod in the cluster can reach any other Pod on any port, with no isolation at all.

Once you apply any NetworkPolicy that selects a given Pod, that Pod's traffic becomes default-deny in whichever direction the policy covers (ingress, egress, or both) — only traffic explicitly allowed by a matching rule gets through, everything else is dropped silently. This is where the classic mistake happens: a team applies a default-deny-all policy for security and forgets to explicitly allow egress to port 53 (DNS), which silently breaks every outbound service-discovery lookup the Pod makes, and looks like a random application failure rather than an obvious network block. Also worth knowing: not every CNI plugin enforces NetworkPolicy at all — the object is a no-op on plugins that don't support it, so it's worth confirming what CNI the cluster actually runs before relying on this for isolation.

Reference: Kubernetes docs — Network Policies

Both route external HTTP(S) traffic into the cluster, but Ingress is a single, fairly minimal resource, while the Gateway API is a family of resources that splits routing into layered roles and expresses far more than Ingress ever could without vendor-specific extensions.

Ingress only really standardizes basic host/path routing; anything beyond that — traffic splitting, request rewriting, timeouts, canary rollouts — has to go through controller-specific annotations, which means the same Ingress YAML behaves differently, or not at all, depending on which controller (NGINX, Traefik, HAProxy...) is installed. The Gateway API fixes this by putting those behaviors into typed, portable fields in the spec itself instead of annotations, and by separating concerns across roles: a GatewayClass/Gateway (owned by infrastructure/platform teams, defining the actual listener and load balancer) and HTTPRoute/GRPCRoute objects (owned by application teams, defining how their traffic gets routed) that attach to it. It also natively supports protocols beyond HTTP — gRPC and TCP/UDP routing are first-class, where Ingress needed controller extensions for the same thing. The practical trend worth knowing: several popular Ingress controllers (notably ingress-nginx) have been moving toward deprecation in favor of Gateway API implementations, so new clusters increasingly start on Gateway API rather than migrating to it later.

Reference: Kubernetes docs — Gateway API

An admission controller intercepts a request to the API server — create, update, or delete — after authentication and authorization succeed, but before the object is persisted to etcd, and gets a chance to modify or reject it. This is the layer where cluster-wide policy actually gets enforced, beyond what RBAC alone can express.

Mutating admission webhooks run first and can change the object on its way in — injecting a sidecar container automatically, setting default resource limits a team forgot to specify, adding a required label for cost tracking. Validating webhooks run after all mutations are applied, and can only accept or reject the (now-final) object — enforcing things like "every Deployment must have resource limits" or "no Pod may run as root," without being allowed to fix it themselves. Both are backed by an HTTP service you run yourself (or a policy engine like OPA Gatekeeper or Kyverno, which handle the plumbing for you); if that service is slow or returns something invalid, the API server's failurePolicy setting decides whether the original request is rejected outright or allowed through anyway, which is a real production availability trade-off — a broken webhook with Fail can block every deployment cluster-wide until it's fixed.

Reference: Kubernetes docs — Dynamic Admission Control

Pod Security Standards define three levels of restriction on what a Pod is allowed to do at the security-sensitive layer — running as root, using host networking or host paths, gaining extra Linux capabilities, running privileged containers. privileged allows essentially anything, baseline blocks the most obviously dangerous settings while staying broadly compatible, and restricted enforces hardened best practice (non-root, no privilege escalation, dropped capabilities, seccomp required).

They're enforced by the built-in Pod Security Admission controller, configured per Namespace via labels rather than a separate policy object — pod-security.kubernetes.io/enforce: restricted, for example. It replaced the older PodSecurityPolicy resource, which was removed from Kubernetes because it was notoriously hard to reason about (policies applied based on a confusing precedence order across RBAC bindings). Pod Security Admission can run in three modes simultaneously per level — enforce (reject), audit (log but allow), and warn (return a warning to the client but allow) — which is the practical way teams roll out a stricter standard without breaking things overnight: set the target level to warn first, watch what would have been rejected, then flip to enforce once nothing unexpected shows up.

Reference: Kubernetes docs — Pod Security Standards

The HPA scales out — it changes how many replicas of a Pod are running. The VPA scales up — it changes how much CPU and memory a single Pod's containers request and are allowed to use. They solve different problems and, run against the same metric on the same workload simultaneously, can actively fight each other.

HPA is the right tool when a workload can be replicated — a stateless API server handles more load fine by adding more identical replicas behind a Service. VPA is the right tool when a workload can't easily be replicated, or when the real problem is that requests/limits were simply sized wrong to begin with — a single-instance component that needs more memory doesn't get helped by adding a second replica, it needs a bigger container. VPA can run in Auto mode, which actually evicts and recreates Pods with new resource values (causing a brief disruption each time it adjusts), or Off/recommendation-only mode, which just tells you what it would set, useful for right-sizing requests without letting it touch running Pods automatically. Running HPA on CPU utilization and VPA on the same container's CPU at the same time is a documented anti-pattern — VPA changing the request out from under HPA changes what "100% utilization" even means, and the two can end up scaling in opposing directions.

Reference: Kubernetes docs — Vertical Pod Autoscaling

Topology spread constraints tell the scheduler to distribute Pods evenly across a topology domain — availability zones, racks, or individual nodes — rather than just saying "not on the same node as this other Pod," which is what anti-affinity expresses. Anti-affinity is binary: a node either satisfies the rule or it doesn't. Spread constraints are proportional: they aim for balance and let you say exactly how much imbalance is tolerable.

The key field is maxSkew, which caps the difference between the topology domain with the most matching Pods and the one with the fewest — maxSkew: 1 across 3 zones for 6 replicas means the scheduler won't let any zone end up with more than one extra Pod compared to the least-loaded zone. whenUnsatisfiable decides what happens if that balance can't be achieved: DoNotSchedule makes it a hard requirement (a Pod can go Pending rather than violate it), ScheduleAnyway treats it as a soft preference. Where anti-affinity gets awkward and hard to reason about once you have many replicas and many zones (writing pairwise "not with this label" rules doesn't naturally express "evenly across 3 zones"), a topology spread constraint expresses that directly in one rule — this is generally the newer, preferred tool for the "don't put all my replicas in one failure domain" problem specifically.

Reference: Kubernetes docs — Pod Topology Spread Constraints

Both are ways kube-proxy implements Service routing, and the difference that matters in an interview is how each one scales. iptables mode evaluates Service rules as a linear sequence — every packet potentially walks through a chain that grows with the number of Services, so lookup cost grows roughly linearly with cluster size. IPVS mode uses an in-kernel hash table for backend selection instead, giving effectively constant-time lookups regardless of how many Services exist, which matters a lot once a cluster has hundreds or thousands of Services and iptables rule evaluation starts becoming a measurable source of latency.

Worth knowing for a current answer rather than a stale one: IPVS mode still relies on iptables underneath for some edge cases (source IP rewriting, certain external traffic paths) — it doesn't replace iptables entirely, it just moves the main backend-selection path off it. It's also worth knowing where the ecosystem is heading: IPVS mode itself has been marked for deprecation in favor of a newer nftables mode (the modern, more maintainable successor to raw iptables rules in the Linux kernel), and some CNI plugins like Cilium bypass kube-proxy's rule-based approach altogether in favor of eBPF. The safe framing for an interview: know why IPVS existed (iptables didn't scale), and be aware the ecosystem has kept moving since then rather than assuming iptables-vs-IPVS is still the final word.

Reference: Kubernetes docs — kube-proxy

A native sidecar is a container defined in a Pod's initContainers list but marked with restartPolicy: Always, which changes how it behaves compared to both a regular init container and a plain second container in containers. It gets the ordering guarantees of an init container — it starts before the main application container does — but instead of running to completion and exiting, it keeps running for the Pod's whole lifetime, and Kubernetes handles its shutdown ordering automatically too.

Before this existed, people ran sidecars as an ordinary second entry in containers, which had two real problems: there was no guarantee the sidecar (say, a service mesh proxy) was actually up and accepting traffic before the main container started sending it requests, and on shutdown, Kubernetes had no way to know the sidecar should be the last thing to stop — it might get killed before the main container finished flushing a final request through it. A native sidecar solves both: it starts first and Kubernetes waits for it to be ready before starting the main containers, and on Pod termination it's stopped only after the main containers have exited, so in-flight traffic routed through a proxy sidecar doesn't get cut off mid-request. It's also the reason kubectl now distinguishes readiness of native sidecars separately when reporting Pod status — a Job with a sidecar no longer stays "running" forever just because the sidecar never exits on its own.

Reference: Kubernetes docs — Sidecar Containers

A normal ClusterIP Service gives you one stable virtual IP that load-balances across whichever Pods are currently ready — you talk to "the Service," never to a specific Pod. A headless Service (clusterIP: None) skips the virtual IP entirely and instead makes DNS return the individual IPs of every matching Pod directly, which is what lets each StatefulSet Pod be addressed by its own stable name.

This matters because StatefulSet workloads usually need to talk to a specific member, not "any ready replica" — a database replica needs to reach the specific primary, not whichever Pod a load balancer happens to pick that request. Pairing a StatefulSet with a headless Service is what makes db-0.db-service.namespace.svc.cluster.local resolve to exactly Pod db-0's IP, giving every replica a durable, individually addressable DNS name that survives rescheduling — the actual mechanism behind the "stable network identity" a StatefulSet is known for. A regular ClusterIP Service simply can't express that; load-balancing across replicas is exactly what it's designed to hide from the caller, which is the opposite of what a clustered stateful workload usually needs for peer discovery.

Reference: Kubernetes docs — StatefulSets: Stable Network ID

A ResourceQuota caps the total resource consumption across an entire Namespace — the sum of every Pod's requests and limits, plus object counts like the maximum number of Pods, Services, or PVCs allowed. A LimitRange sets defaults and bounds per individual object within a Namespace — the min/max a single container's requests or limits are allowed to be, and what value to apply if a Pod spec doesn't specify one at all.

They're usually deployed together because a ResourceQuota alone has a sharp edge: once a Namespace has a quota defined for CPU or memory, the API server requires every Pod created in that Namespace to explicitly specify requests and limits for that resource — Pods that don't will be rejected outright. A LimitRange fixes this by supplying sane defaults automatically, so teams that forgot to set requests/limits don't suddenly get every deployment blocked the moment a quota is introduced. In short: LimitRange keeps any one Pod from being absurdly oversized or completely unsized, ResourceQuota keeps the Namespace as a whole from consuming more than its fair share of the cluster.

Reference: Kubernetes docs — Resource Quotas

RollingUpdate, the default, replaces old Pods with new ones gradually, controlled by maxSurge (how many extra Pods can exist above the desired count during the update) and maxUnavailable (how many can be missing) — the goal is zero downtime, at the cost of briefly running both old and new versions side by side. Recreate is blunt by comparison: it terminates every existing Pod first, and only then starts the new ones, guaranteeing a gap with zero Pods running.

Recreate sounds strictly worse, but it's the correct choice in specific situations where running old and new versions simultaneously would actively cause harm — a schema-breaking database migration where the old and new application versions can't safely operate against the data at the same time, or a workload that can't tolerate two versions writing to the same non-shareable volume concurrently (a ReadWriteOnce volume that only one Pod can mount at a time would make RollingUpdate fail regardless, since the new Pod can't mount the volume until the old one releases it). The trap: teams sometimes pick Recreate by default "to be safe" without realizing it introduces guaranteed downtime on every single deploy, when the actual problem it solves (incompatible concurrent versions) usually doesn't apply to their workload.

Reference: Kubernetes docs — Deployment Strategy

CrashLoopBackOff is a status, not a root cause — it just means the container keeps exiting and the kubelet is waiting progressively longer (10s, 20s, 40s... up to 5 minutes) before retrying. Empty logs almost always mean the container is crashing before the application ever writes a log line, so log-first debugging doesn't work here and you need to start elsewhere.

The order that actually gets to an answer:

1. kubectl describe pod <name> and read the Events section first. This is where OOMKilled, a failed probe, or an image-pull failure shows up before the app ever gets a chance to log anything.
2. kubectl logs <name> --previous. This pulls logs from the previous crashed instance, not the current restart attempt — the current attempt may not have produced any output yet, but the last one usually printed something on its way down.
3. Check the exit code in the "Last State" section of describe. 137 means SIGKILL — almost always OOMKilled or a manual kill. 1 or another app-specific code means the process exited on its own, which points back at application logic or misconfiguration, not Kubernetes.
4. Rule out a misconfigured liveness probe. If the app is genuinely healthy but slow to start, an aggressive liveness probe will kill it before it's ready, over and over, which looks identical to an application crash from the outside. Temporarily removing the liveness probe (or checking initialDelaySeconds) isolates this quickly.
5. If none of that explains it, attach an ephemeral debug container (kubectl debug) to get a shell inside the Pod's namespaces without needing to modify or restart it — essential for distroless or scratch images that ship no shell at all.

The trap: trying to kubectl exec or port-forward into a Pod that's actively crash-looping. It's not a stable debugging target — fix or at least pause the crash first (e.g. by overriding the entrypoint to sleep infinity in a throwaway copy of the Pod), rather than fighting a moving target.

Reference: Kubernetes docs — Debug Running Pods

The 4th replica stays Pending forever, and the other three keep running normally. A requiredDuringSchedulingIgnoredDuringExecution anti-affinity rule is a hard constraint — the scheduler will not place two matching Pods on the same node no matter how much capacity that node has, so with only 3 nodes available there is nowhere left for a 4th Pod to legally go.

kubectl describe pod on the pending replica will show a scheduling failure in Events, something like "0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules." The Deployment itself reports as not fully available (e.g. 3/4 ready) but doesn't error out or roll back — it just sits there with one replica permanently unschedulable until either a 4th node joins the cluster or the anti-affinity rule is relaxed to preferredDuringSchedulingIgnoredDuringExecution, which the scheduler will try to honor but won't block on if it can't. This is a good example of why "required" affinity/anti-affinity rules need to be sized against actual cluster capacity, not just written to express intent — the cluster autoscaler can help here by adding a node in response to the pending Pod, but only if it's configured and the cloud account has room to scale.

Reference: Kubernetes docs — Inter-pod Affinity and Anti-affinity

A stalled rolling update almost always means the new Pods aren't passing their readiness probe, because a Deployment's RollingUpdate strategy won't terminate more old Pods or bring up more new ones than maxUnavailable/maxSurge allow until the Pods it has already created become Ready — it's the Deployment controller behaving correctly, just blocked on unhealthy new Pods.

Start with kubectl get pods -l app=<name> to see which of the new-revision Pods aren't Ready, then kubectl describe pod on one of them. Common causes, roughly in order of likelihood: the new image genuinely fails its readiness probe (bad build, missing config, a dependency the old version didn't need); a ConfigMap or Secret the new revision references doesn't exist yet; a resource request that doesn't fit on any available node, leaving new Pods Pending instead of just unready; or an image pull failure on a private registry (ImagePullBackOff). kubectl rollout status deployment/myapp will keep reporting "waiting" rather than timing out unless you've set progressDeadlineSeconds, so a genuinely stuck rollout can sit like this indefinitely with no automatic alert unless that's configured.

Once you've confirmed it's broken rather than just slow, the fastest recovery is usually kubectl rollout undo deployment/myapp to roll back to the last known-good revision while you fix the new one offline, rather than debugging live against production traffic.

Reference: Kubernetes docs — Performing a Rolling Update

Base64 is an encoding, not encryption — it has no key, and decoding it is a one-line command, not a cryptographic attack. So by default, anyone who can read a Secret object through the API (subject to RBAC), or anyone with direct read access to the etcd data files or an etcd snapshot backup, can recover every Secret's plaintext value trivially. "It's stored as a Secret" provides organizational structure, not confidentiality, unless you add real controls on top of it.

The practical hardening, layered:

• Encryption at rest. Configure an EncryptionConfiguration on the API server so Secret data is actually encrypted before it's written to etcd — this closes the "raw etcd snapshot or disk access" hole specifically. Without it, an etcd backup file is just as readable as the live API, which is easy to forget when backup storage gets a different security review than the cluster itself.
• RBAC scoped tightly. get/list/watch on secrets should be granted per-Namespace to only the identities that need it, not handed out with broad view roles that happen to include Secrets incidentally. Remember RBAC is purely additive — a single overly broad ClusterRoleBinding undoes every narrower Role you defined elsewhere.
• Restrict etcd access directly: TLS on client and peer communication, no direct network exposure, and treat etcd snapshot backups as being just as sensitive as the live cluster, with the same access controls and encryption applied to backup storage.
• For genuinely high-sensitivity secrets, consider an external secrets manager (Vault, cloud KMS-backed secret stores) with Kubernetes only holding a short-lived reference, rather than the actual value, at all.

Reference: Kubernetes docs — Encrypting Confidential Data at Rest

The goal is to get the Pod cleanly stopped and its replacement started elsewhere with the same PVC reattached, without ever having two Pods writing to the same volume at once and without skipping the application's own shutdown sequence. This is one of the places where "just delete the Pod" is actively dangerous.

1. Set (or confirm) a PodDisruptionBudget for the StatefulSet before draining anything, so the drain can't take out more replicas than the application can tolerate simultaneously — for a 3-node quorum-based system, that's very likely minAvailable: 2 or equivalent, not "however many happen to survive."
2. Give the app a real preStop hook and grace period, not just the default terminationGracePeriodSeconds. For anything with in-memory state or an active connection pool, an abrupt SIGKILL after too short a grace period risks a partial write; a preStop hook can trigger a clean shutdown sequence (flush buffers, deregister from a peer group, finish in-flight writes) before the container actually stops.
3. kubectl drain <node> --ignore-daemonsets, which cordons the node first (so nothing new schedules there) then evicts Pods respecting the PDB — it won't proceed past the budget's floor, and will wait rather than force through it.
4. Confirm the PV's access mode and storage backend support detach/reattach cleanly. For ReadWriteOnce volumes on typical cloud block storage (EBS, PD), Kubernetes handles the detach-from-old-node / attach-to-new-node sequencing itself as part of eviction and rescheduling — but if the PVC is stuck in Terminating afterward, that's almost always a leftover finalizer blocking cleanup, which should only be removed manually once you've confirmed there's no still-active I/O against that volume, since force-removing a finalizer while a mount is genuinely active is exactly how you get corruption.
5. Let the StatefulSet's ordered guarantees do their job — the replacement Pod comes back with the same ordinal and the same PVC by design, so as long as steps 1–4 didn't force anything, this step is mostly "don't fight it by manually deleting PVCs or reordering things yourself."

Reference: Kubernetes docs — Safely Drain a Node

Flapping almost always comes down to the HPA reacting to noise in a metric rather than a genuine, sustained change in load — and it's a metrics/tuning problem, not evidence that autoscaling itself is broken.

The usual culprits, and what to check for each: CPU or memory requests set unrealistically low or high, which makes utilization percentage swing wildly for the same absolute load — the HPA targets a percentage of the request, so a badly sized request makes the percentage noisy even when raw usage barely moved. A metric with genuinely bursty, short-lived spikes (a periodic batch job, a cron-triggered cache warm, a noisy neighbor Pod on the same node skewing node-level metrics) that crosses the target threshold for a few seconds at a time. Too-aggressive scaling behavior config — by default the HPA has a stabilization window that's meant to prevent exactly this, but if it's been shortened, or the scale-up and scale-down policies allow large steps, small metric wobbles turn into visible replica churn. A downstream effect of the scaling itself: each new Pod takes time to become Ready, and if the metric being scaled on doesn't account for not-yet-ready Pods properly, the HPA can perceive load as still high right after scaling up, and scale up again before the new Pods have had a chance to absorb any traffic.

The fix is almost never "give up on autoscaling" — it's tightening the target and behavior config: widen the stabilization window on scale-down specifically (this is the safe direction to be conservative on), right-size the underlying resource requests first since a bad target percentage undermines everything else, and consider scaling on a steadier signal (e.g. requests-per-second via a custom metric) instead of CPU if the workload's CPU usage is inherently bursty by nature rather than by misconfiguration.

Reference: Kubernetes docs — HPA: Support for scaling behavior

Use an ephemeral debug container — a temporary container that Kubernetes attaches directly into a running Pod's existing namespaces (network, process, and optionally filesystem), without restarting the Pod or needing the original image to contain any tooling at all. This is the purpose-built answer to distroless and scratch images specifically; it didn't really have a clean solution before ephemeral containers existed.

--target=app attaches the debug container to share the process namespace of the named container specifically, so from inside the debug container you can see the target's running processes (ps aux shows the real app process, not just the debug container's own), inspect open files via /proc/<pid>/, and reach the same network namespace — meaning curl localhost:8080 from the debug container talks to the app exactly as if you'd shelled into it directly, even though the app's own image has no shell to exec into in the first place.

Two other tools solve adjacent but different problems worth distinguishing: kubectl debug --copy-to creates a full copy of the Pod with a modified spec (useful when you need to change the entrypoint or resource limits to reproduce something, without touching the live Pod at all), and kubectl debug node/<name> attaches a debug container in the host's namespace, useful when the problem is node-level rather than Pod-level — a full disk, a kernel setting, a misbehaving daemon — not something you can see from inside any Pod.

Reference: Kubernetes docs — Debugging with Ephemeral Debug Container

No single mechanism gives you real multi-tenancy — it's a combination, each covering a different failure mode, and the common wrong answer is reaching for just one (usually "give each team a Namespace") and assuming that's isolation.

• Namespaces as the organizing boundary — one per team or environment — but on their own they don't stop cross-namespace network traffic, don't limit resource consumption, and don't restrict what a Pod is allowed to do at the kernel level. Treat them as the scope everything else attaches to, not as isolation itself.
• RBAC, scoped per-Namespace with Roles and RoleBindings, so a team can manage its own workloads but can't read or modify another team's objects, Secrets, or RBAC rules — with least privilege as the default rather than something bolted on afterward.
• ResourceQuotas and LimitRanges per Namespace, so one team can't consume the whole cluster's CPU, memory, or object count and starve everyone else. A quota without sane default requests/limits via a LimitRange still lets a single misconfigured Pod request unreasonable amounts, so the two are usually applied together.
• NetworkPolicies, generally starting from a default-deny baseline per Namespace with explicit allow rules for the traffic that's actually expected — this is what actually stops one team's Pods from reaching another team's Pods over the network, which Namespaces alone don't do.
• Pod Security Admission, enforcing the restricted (or at minimum baseline) Pod Security Standard per Namespace, to stop workloads from running as root, using host networking, or escalating privileges in ways that could affect the node — and by extension, every other tenant's Pods scheduled on that same node.
• For genuinely hostile or high-stakes multi-tenancy — untrusted third-party code, strict compliance boundaries — Namespace-level isolation inside one cluster is often judged insufficient regardless of how well the above is configured, and separate clusters per tenant become the actual answer; it's worth naming that ceiling rather than implying config alone always solves it.

Reference: Kubernetes docs — Multi-tenancy

NotReady means the control plane hasn't received a recent heartbeat from that node's kubelet — it does not mean the Pods on it have actually stopped. The API server simply has stale information; the Pods could still genuinely be running fine on a node whose kubelet is just failing to report in, or they could already be dead and Kubernetes hasn't found out yet. Both are real possibilities and the debugging path has to account for that ambiguity rather than assume either one.

The first move is kubectl describe node <name> to read the actual condition and reason — NotReady can stem from the kubelet process being down, the container runtime (containerd/CRI-O) crashing under the kubelet, disk pressure, memory pressure, or a networking partition between the node and the control plane, and each has a different fix. If you have direct node access, systemctl status kubelet and journalctl -u kubelet (and the same for the container runtime) usually shows which of those it is. Kubernetes itself has a built-in grace period here: by default, if a node stays NotReady for 5 minutes, the control plane marks its Pods' status as Unknown and begins scheduling replacements elsewhere (assuming the workload is managed by a controller like a Deployment) — but crucially, it can't force-stop the original Pods on the unreachable node, because it has no way to reach it. This is exactly the scenario where you can temporarily end up with the same Pod appearing to run in two places — the original, unreachable one, and a freshly scheduled replacement — until the node either recovers and the kubelet reconciles reality, or someone confirms the node is truly gone and deletes it, which is what actually releases those stale Pod objects.

The trap: assuming NotReady automatically means the workload is down and taking manual action (killing Pods, forcing failover) before confirming whether the node is actually unreachable versus genuinely dead — those two situations call for different responses, and guessing wrong risks exactly the split-brain scenario described above.

Reference: Kubernetes docs — Node Status

Cluster DNS failures are rarely "DNS is broken" — they're almost always "something specific about these Pods' path to CoreDNS is broken," which is why the fact that it's only some Pods is the most useful clue in the question, not a detail to skip past.

Work from what's shared versus what's different between the failing and working Pods:

• Same node vs different nodes. If every failing Pod is on one node, or one just recovered from NotReady, suspect stale conntrack entries for UDP DNS traffic on that node specifically — a known failure mode where the kernel's connection-tracking table holds onto a mapping to a CoreDNS Pod that no longer exists (rescheduled elsewhere), and new DNS queries keep getting silently routed nowhere until the stale entry expires or is manually cleared.
• CoreDNS Pod health itself. kubectl get pods -n kube-system -l k8s-app=kube-dns — CoreDNS can be Running but not Ready (its own readiness probe checks that it can reach the Kubernetes API to serve cluster DNS records; if that connection is unhealthy, CoreDNS logs still waiting on: "kubernetes" in a loop and never signals ready, which quietly removes it as a valid endpoint without the Pod ever going into an obviously broken status).
• NetworkPolicy. A default-deny-egress policy applied to only some Namespaces (or added recently) that forgot to explicitly allow UDP/TCP port 53 to kube-system will break DNS for exactly the Pods it covers and nothing else — this is common enough that it should be checked early, not last.
• Resolver config on the Pod itself. kubectl exec into a failing Pod and check /etc/resolv.conf — a custom dnsPolicy or dnsConfig on just that workload's Pod spec can point it at the wrong nameserver entirely, independent of whether CoreDNS itself is healthy.
• Direct test from inside the cluster, isolating the DNS layer from the application: kubectl run -it --rm dnsutils --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- nslookup kubernetes.default run from both a working and a failing Namespace/node narrows down whether it's node-level, Namespace-level, or Pod-spec-level.

Reference: Kubernetes docs — Debugging DNS Resolution

The core idea is that the control plane and the workloads it manages are largely decoupled — a properly configured application keeps serving traffic through a control plane upgrade, because kubelets keep running Pods locally even if they briefly can't reach the API server, and a well-designed rollout only ever touches one plane component or one node at a time.

The rough shape of a real plan:

1. Read the version's release notes and API deprecations first, not last. Kubernetes only supports upgrading one minor version at a time (e.g. 1.30 → 1.31, not 1.30 → 1.32 directly), and any manifests using an API version removed in the target release need to be migrated before upgrading, or they'll simply fail to apply afterward.
2. Upgrade the control plane components first, one at a time, in order: kube-apiserver, then controller-manager and scheduler, then etcd if it needs a version bump too. Managed platforms (EKS, GKE, AKS) handle this for you with the tenant-visible impact usually limited to a brief API server unavailability window, not a workload outage — this is exactly why a properly configured application shouldn't notice: existing Pods keep running under the old kubelet regardless of whether the API server is briefly unreachable.
3. Upgrade kubelets node by node, using the same drain discipline as any node maintenance — cordon, respect PodDisruptionBudgets, wait for Pods to reschedule cleanly elsewhere, upgrade, uncordon. This is where PDBs defined ahead of time matter most: without them, upgrading multiple nodes at once could evict more replicas of a service than it can tolerate simultaneously.
4. Roll node pools rather than upgrading nodes in place where the platform supports it (common on managed Kubernetes) — provision new nodes on the target version, drain and cordon old ones gradually, delete old nodes once workloads have migrated. This avoids ever running a kubelet mid-upgrade in an inconsistent state.
5. Validate before declaring done: confirm no Pods are stuck Pending due to a capacity gap introduced mid-rollout, and check for any workload still relying on a now-removed API version that slipped through step 1.

The trap: treating "the control plane is highly available" as the same guarantee as "workloads won't notice." HA control plane prevents an outage, but a careless kubelet rollout that ignores PDBs can still take down more replicas than intended even with a perfectly healthy control plane throughout.

Reference: Kubernetes docs — Upgrading kubeadm clusters

The risk is a brief window where two processes both believe they own the same PersistentVolume — the old container, if it's actually still running on a node the API server just lost contact with, and the newly created replacement Pod that a force-delete tells Kubernetes to treat as already gone. A normal kubectl delete pod waits for the kubelet to confirm the container has actually stopped before Kubernetes considers it terminated; --grace-period=0 --force skips that confirmation entirely and just deletes the object from the API, which is precisely why it's a forced action and not the default.

This is dangerous specifically because "unresponsive" doesn't necessarily mean "actually stopped." If the node lost network connectivity to the control plane but the container process itself is still alive and still has the volume mounted, force-deleting the Pod object doesn't kill that process — it just tells Kubernetes to forget about it and free up the identity for a replacement. If the replacement Pod then gets scheduled and mounts the same ReadWriteOnce volume from a different node, you can end up with two live processes writing to the same underlying storage simultaneously, with no coordination between them — a classic split-brain scenario for a database, and one of the more damaging ways to lose or corrupt data in Kubernetes.

The safer sequence:

1. First confirm the node is actually unreachable versus the process truly dead — kubectl describe node and, if possible, direct access to the node, rather than assuming from Pod status alone.
2. If the node is confirmed dead (hardware failure, terminated instance) and will never come back, force-delete is legitimate — there's no process left to conflict with.
3. If the node might still be alive and just partitioned from the control plane, fence it first — cordon it, and if you have out-of-band access (cloud provider API, IPMI, a hypervisor), actually power it off or confirm the process is stopped before force-deleting the Pod object, so the replacement can't race against a survivor.
4. Prefer letting Kubernetes's own timeout path run its course when there's no urgency — the default 5-minute NotReady grace period followed by automatic Pod-status transition to Unknown exists specifically to avoid a human jumping straight to force-delete before there's any real confirmation the old Pod is gone.

Reference: Kubernetes docs — Force Delete StatefulSet Pods