Kubernetes Interview Questions

A Pod is the smallest deployable unit in Kubernetes: one or more containers that share one network namespace (one IP address, one port space) and can share storage volumes. Containers in the same Pod talk to each other over localhost.

Kubernetes schedules Pods rather than bare containers because some workloads need more than one container acting as a single unit, such as a main app plus a sidecar that ships logs or proxies traffic. Putting them in one Pod guarantees they land on the same node, start and stop together, and can share files without extra networking.

Most Pods run a single container, but the Pod is the abstraction Kubernetes uses for scheduling, networking and lifecycle. It never reasons about a container on its own.

The common trap is describing a Pod as "just a wrapper around one container". That is true in most deployments, but it misses why the abstraction exists, which is usually the real follow-up question.

Reference: Kubernetes docs - Pods

A ReplicaSet keeps a set number of identical Pods running. If one dies, it starts another. A Deployment sits one layer above and manages ReplicaSets for you, adding rolling updates, rollbacks and revision history.

In practice you almost never create a ReplicaSet directly. When you change a Deployment's Pod template (a new image tag, a changed environment variable), the Deployment creates a new ReplicaSet and gradually moves Pods from the old one to the new one. It does not edit Pods in place.

The old ReplicaSet is kept at zero replicas so kubectl rollout undo has something to go back to. Scaling a Deployment does not create a new ReplicaSet. Only a change to the Pod template does.

Reference: Kubernetes docs - Deployments

A Service gives a stable IP address and DNS name to a group of Pods. Pods are temporary: every restart, reschedule or rolling update gives a Pod a new IP, so tracking Pod IPs yourself is not practical.

A Service selects Pods by label and keeps an up-to-date list of the healthy ones as Pods start, stop, and pass or fail readiness checks. Clients talk to the Service's stable address, and traffic is spread across whichever Pods are ready right now.

This "talk to the Service, not the Pod" idea is what makes rolling updates and self-healing work without breaking the things that depend on your app.

Reference: Kubernetes docs - Service

Both store configuration outside your container image and inject it into Pods as environment variables or mounted files. The difference is intent. A ConfigMap is for non-sensitive values like feature flags or log levels. A Secret is for sensitive values like passwords, API keys and TLS certificates.

The trap: Secrets are base64-encoded, not encrypted, by default. Base64 is a reversible encoding, not a cipher. Anyone who can read the Secret through the API, or read etcd directly, can decode it with one command.

To really protect Secrets you need encryption at rest on the cluster and tight RBAC so only the Pods and people who need a Secret can read it. Treat "it is a Secret object" as labelling, not as a security control by itself.

Reference: Kubernetes docs - Secrets

A Namespace divides one cluster into several virtual clusters. It gives you a scope for names, RBAC rules, resource quotas and NetworkPolicies. Two Deployments both called api can live side by side if they are in different Namespaces.

Use Namespaces to separate environments (staging and production on shared infrastructure), to separate teams on a shared cluster, or to apply different RBAC and quota rules to different parts of a system.

They are not a hard security boundary by default. A Pod in one Namespace can usually still reach a Service in another unless a NetworkPolicy blocks it. So Namespaces alone are not enough for strong isolation between teams that do not trust each other.

Reference: Kubernetes docs - Namespaces

A DaemonSet makes sure one copy of a Pod runs on every node (or every node matching a selector), and adds a Pod automatically when a new node joins.

A Deployment does not care which node a Pod lands on. It only wants N replicas running somewhere. A DaemonSet cares about node coverage, so you use it for per-node tools like a log shipper, a metrics collector or a network plugin, where you need "one on every node".

If you scaled a Deployment to match your node count instead, the scheduler could still put two Pods on one node and none on another. A DaemonSet guarantees the one-per-node rule directly.

Reference: Kubernetes docs - DaemonSet

etcd is the distributed key-value store that holds the whole state of a Kubernetes cluster. Every object you can kubectl get, from Pods and Deployments to Secrets and RBAC rules, lives there. The API server is the only component that talks to etcd directly. The scheduler, controllers and kubelets all go through the API server.

Because etcd holds everything, including Secrets, it is a high-value target. Anyone with direct etcd access can read Secret values regardless of RBAC unless encryption at rest is on.

etcd runs as a Raft cluster (usually 3 or 5 members). If more than half the members are lost, the cluster loses quorum and the API stops working even if every node is healthy. That is why regular etcd backups are treated as non-negotiable in production.

Reference: Kubernetes docs - Cluster components

kubectl create is imperative. It creates a resource from a manifest and fails if the resource already exists. kubectl apply is declarative. It creates the resource if it is missing, or compares your file to the live object and patches only what changed if it exists.

This matters because apply is what you use in CI/CD and GitOps. You can run the same command again and again against version-controlled YAML and the cluster converges to match the file, instead of erroring on the second run.

apply also stores the last applied configuration, which is how it can tell your changes apart from changes made by someone or something else. create is mostly for quick one-off tests.

Reference: Kubernetes docs - Managing resources

kube-proxy runs on every node and turns a Service's stable virtual IP into real traffic that reaches one of the Service's backend Pods. A Service IP is not attached to any network card. It only works because kube-proxy watches Services and endpoints and programs the node's networking so traffic to that IP is forwarded to a Pod.

The application never knows kube-proxy exists. The redirection happens at the operating system networking layer.

Historically it used iptables rules, with an IPVS mode for large clusters, and newer versions add an nftables mode. Some network plugins, such as Cilium, replace kube-proxy completely with eBPF. The idea to remember: Services are a Kubernetes abstraction, and kube-proxy makes that abstraction real on the wire.

Reference: Kubernetes docs - kube-proxy

A CRD lets you add your own object types to the Kubernetes API. After you create one, kubectl get certificates or kubectl get databases works just like kubectl get pods, with the same API style, the same verbs and the same RBAC.

On its own, a CRD is only a schema. The API server will store and serve those objects, but nothing acts on them. You also need a controller (often called an operator) that watches the objects and does real work, such as creating a database or requesting a TLS certificate.

CRD plus controller is what people usually mean by "an operator". The CRD holds the desired state and the controller moves the cluster toward it, the same pattern Kubernetes uses internally for Deployments.

Reference: Kubernetes docs - Custom Resources

An image is a read-only bundle of files (your code, dependencies and filesystem layout), built once and stored in a registry. A container is a running instance of that image: the image plus a thin writable layer, plus the process, namespaces and resource limits the runtime gives it.

Think of a class and an object, or a recipe and a meal. One image can start many containers at once, each with its own writable layer, and none of them change the image.

This is why restarting a container throws away anything it wrote to its own filesystem unless that path is on a volume. It is also why images are treated as immutable: you do not fix a running container by editing it, you build a new image and roll it out.

Reference: Kubernetes docs - Images

A sidecar is a second container in the same Pod that supports the main application without being the thing that does the app's job. Because containers in a Pod share a network and can share volumes, a sidecar can intercept traffic, tail log files or sync data without the main app knowing.

Common examples: a service mesh proxy such as Envoy that handles mTLS and traffic policy, a log shipper that tails a shared volume, or a container that refreshes config from an outside source.

A classic sidecar runs for the whole life of the Pod. That is what separates it from an init container, which runs once and exits. Starting and stopping sidecars in the right order used to be awkward, which is why native sidecar support was added (see the native sidecar question later in this page).

Reference: Kubernetes docs - Sidecar containers

Labels are key-value pairs attached to objects, such as app: api or env: prod. Selectors are queries that pick objects by their labels. Kubernetes uses this loose matching instead of hard links between objects.

Almost everything is connected this way. A Service finds its Pods by selector. A Deployment's ReplicaSet owns the Pods that match its selector. A NetworkPolicy, PodDisruptionBudget and topology spread constraint all pick Pods by label too.

The trap is a typo or mismatch. If a Service selector says app: api and the Pods are labelled app: apis, the Service has no endpoints and traffic goes nowhere, with no error message. Checking labels is usually the first step when "the Service exists but nothing answers".

Reference: Kubernetes docs - Labels and selectors

Labels identify and group objects, and Kubernetes can select on them. Annotations hold extra information that Kubernetes does not use for selection, such as a build number, a contact email, a link to a runbook, or settings for a tool like an Ingress controller.

The easy rule: if you need to query or select by it, use a label. If it is just metadata for humans or tools to read, use an annotation.

Labels have tight limits on length and allowed characters because they are indexed. Annotations can be much larger and hold almost any text, including JSON.

A common mistake is putting a long or frequently changing value in a label, such as a timestamp or a git message. That adds noise to every selector and can slow down tools that watch labels.

Reference: Kubernetes docs - Annotations

The kubelet is the agent that runs on every node. It watches the API server for Pods assigned to its node, then makes the container runtime start, stop and restart containers so reality matches what the Pod spec says.

It also runs the liveness, readiness and startup probes, mounts volumes, reports the node's health and capacity, and sends Pod status back to the API server. If the kubelet stops reporting, the control plane eventually marks the node NotReady.

Notice what it does not do. It does not decide where Pods go (that is the scheduler), and it does not create replacement Pods (that is the controllers). It only looks after the Pods on its own node.

The kubelet can also run static Pods straight from files on disk, which is how kubeadm starts the control plane itself.

Reference: Kubernetes docs - kubelet

The scheduler picks a node for each new Pod that does not have one yet. It does not start the Pod. It only writes the chosen node name into the Pod, and the kubelet on that node does the rest.

To choose, it first filters out nodes that cannot run the Pod (not enough CPU or memory, a taint the Pod does not tolerate, a failed nodeSelector or affinity rule). Then it scores the remaining nodes and picks the best one, for example the one with the most free resources or the best spread.

If no node passes the filter, the Pod stays Pending, and kubectl describe pod shows why in the events. Understanding that the scheduler only decides placement helps you debug: a Pod stuck Pending is a scheduling problem, while a Pod stuck ContainerCreating is already on a node and is a kubelet or runtime problem.

Reference: Kubernetes docs - Scheduler

A controller is a program that watches the cluster's state and works to make the actual state match the desired state you declared. This repeating cycle of observe, compare, act is called the reconciliation loop.

For example, you declare 3 replicas in a Deployment. The ReplicaSet controller counts the matching Pods. If it sees 2, it creates one. If it sees 4, it deletes one. It does this continuously, not just once, which is why Kubernetes heals itself when a Pod dies.

The built-in controllers run inside the kube-controller-manager. Operators and your own custom controllers use the same pattern for custom resources.

The key idea for interviews: Kubernetes is declarative. You describe what you want, and controllers keep working toward it. You do not give step-by-step commands.

Reference: Kubernetes docs - Controllers

The kube-apiserver is the only component that exposes the Kubernetes API and the only one that talks to etcd. Every other part of the cluster, including kubectl, the scheduler, the controllers and the kubelets, talks to the API server instead of to each other.

When a request arrives, it goes through authentication (who are you), authorization (are you allowed, usually RBAC), and admission control (should this be changed or rejected), and only then is the object saved to etcd.

Because everything flows through it, the API server is where you enforce security and policy, and it is also a single point to protect. Running several API server replicas behind a load balancer is the normal way to make the control plane highly available.

Reference: Kubernetes docs - API server

The control plane is the cluster's brain. It makes decisions and stores state: the API server, etcd, the scheduler and the controller manager. Worker nodes are where your application Pods actually run. Each one runs a kubelet, a container runtime and kube-proxy (or an equivalent).

On a managed service such as EKS, GKE or AKS, the provider runs the control plane for you and you only manage the worker nodes.

An important point: worker nodes keep running their Pods even if the control plane goes down for a while. You just cannot make changes, and nothing replaces Pods that fail until the control plane returns.

By default, control plane nodes carry a taint so ordinary workloads are not scheduled on them.

Reference: Kubernetes docs - Cluster architecture

A Pod's status.phase is a short summary of where it is in its life:

The trap: Running does not mean healthy or ready. A Pod can be Running and still failing its readiness probe, or crash-looping between restarts. Also, CrashLoopBackOff and ImagePullBackOff are not phases. They are container states shown in the STATUS column of kubectl get pods.

For real health, look at the Ready condition and the container states, not just the phase.

Reference: Kubernetes docs - Pod lifecycle

Helm is the package manager for Kubernetes. A chart is a bundle of templated Kubernetes manifests plus default settings. You install it with your own values, and Helm renders the templates and applies them as one unit called a release.

It solves two problems. First, a real app needs many manifests (Deployment, Service, Ingress, ConfigMap) and copying them between environments is error-prone. Templates with a values.yaml let you reuse one chart for dev, staging and production. Second, Helm tracks each release and its revisions, so you can upgrade and roll back as one action.

Helm 3 has no server-side component (the old Tiller was removed), and it stores release history as Secrets in the release's Namespace.

Reference: Helm docs - Using Helm

A ServiceAccount is an identity for a process running in a Pod, so the Pod can authenticate to the Kubernetes API. Kubernetes has no user objects for humans. People come from outside, through certificates, OIDC or a cloud identity provider. ServiceAccounts are the one identity type Kubernetes manages itself.

Every Namespace has a default ServiceAccount, and Pods use it unless you say otherwise. You grant it permissions with a Role and a RoleBinding like any other subject.

A good habit is to create a dedicated ServiceAccount per app with only the permissions it needs, and to set automountServiceAccountToken: false for Pods that never call the API. The common mistake is leaving everything on default and then giving default broad access, which grants those rights to every Pod in the Namespace.

Reference: Kubernetes docs - Service accounts

A StorageClass describes a type of storage a cluster can provide, such as fast SSD, cheap HDD or a network file share. It names the provisioner (usually a CSI driver) and its parameters. When a PersistentVolumeClaim asks for that class, Kubernetes creates a matching volume automatically. This is called dynamic provisioning.

It also sets useful behaviours: the reclaimPolicy (whether the disk is deleted or kept when the claim is deleted), whether volumes can be expanded, and the volumeBindingMode.

Clusters usually mark one class as the default, so a PVC that does not name a class still gets storage. If a PVC sits Pending, a missing default class or a wrong class name is a very common cause.

Reference: Kubernetes docs - Storage classes

A kubeconfig file (by default ~/.kube/config) tells kubectl how to connect to clusters. It holds three things: clusters (API server address and CA), users (credentials), and contexts. A context ties one cluster, one user and an optional default Namespace together under a name.

The current context decides which cluster your commands hit. That makes it easy to switch between, say, dev and prod with one command.

The danger is running a destructive command against the wrong context. Many teams show the current context in their shell prompt, and use separate kubeconfig files or read-only credentials for production to avoid mistakes.

Reference: Kubernetes docs - Configure access to multiple clusters

An emptyDir is a temporary directory created when a Pod is assigned to a node. It starts empty, all containers in the Pod can read and write it, and it is deleted forever when the Pod is removed from the node. A PersistentVolume outlives the Pod.

Use emptyDir for scratch space, caches, or sharing files between containers in the same Pod, such as an app writing logs that a sidecar ships. It survives container restarts inside the Pod, but not Pod deletion or rescheduling.

You can back it with memory (medium: Memory), which is fast but counts against the container's memory limit.

The trap: storing data you care about in an emptyDir. It looks like a volume, so it feels safe, but a rescheduled Pod loses everything.

Reference: Kubernetes docs - Volumes

They are three levels of exposure, each built on the one before.

A NodePort opens that port on every node, even nodes with no matching Pod. kube-proxy then forwards the traffic to a Pod. People often expect it to open only where the workload runs.

A LoadBalancer Service is built on top of a NodePort. The cloud provider's controller creates an external load balancer that forwards to it. On bare-metal clusters with no cloud integration, a LoadBalancer Service stays in <pending> unless something like MetalLB fulfils it.

Reference: Kubernetes docs - Service

Use an Ingress when you want to route HTTP and HTTPS traffic for many services through one entry point, based on hostname or URL path, with TLS handled in one place. That avoids giving every service its own cloud load balancer.

A LoadBalancer Service gives you one external IP per Service, which gets expensive fast, and it works at layer 4, so it knows nothing about paths or hostnames. An Ingress holds routing rules (api.example.com goes here, example.com/blog goes there), and an Ingress controller such as NGINX, Traefik or HAProxy does the routing, usually behind a single LoadBalancer.

The trade-off is that you now depend on the controller being installed and healthy. Ingress is also not the right tool for raw TCP or UDP traffic such as databases.

The Gateway API is the newer, more expressive design from Kubernetes SIG Network and is increasingly used in new clusters, but Ingress still runs in a huge number of existing ones.

Reference: Kubernetes docs - Ingress

A liveness probe asks "should this container be restarted?". A readiness probe asks "should this Pod receive traffic right now?". They cause different actions when they fail, and people mix them up often.

If liveness fails, the kubelet kills and restarts the container, assuming it is stuck. If readiness fails, the Pod is removed from the Service's endpoints and keeps running, it just stops getting new traffic until the probe passes again.

A startup probe handles slow-starting apps. While it is still failing, liveness and readiness checks are paused, so a slow boot is not killed too early.

The classic trap is a liveness probe that checks a database. When the database has an outage, every Pod fails liveness at once and restarts in a loop. Restarting the app does not fix the database, so you turn one outage into two. Keep liveness checks cheap and local. Put dependency checks in the readiness probe.

Reference: Kubernetes docs - Liveness, readiness and startup probes

Nothing happens to the running Pods. A rolling update starts only when .spec.template changes, not on any other field and not just because you ran apply.

If you change replicas, add an annotation outside the Pod template, or re-apply an identical file, the Deployment controller reconciles but does not create a new ReplicaSet. It watches the content of the template itself.

That is why kubectl scale does not add an entry to kubectl rollout history. It is also why kubectl rollout restart exists. It patches an annotation inside the Pod template, which counts as a template change and triggers a real rolling restart. That is handy when you want Pods to pick up a changed ConfigMap without changing the image.

Reference: Kubernetes docs - Rolling update

A request is what the scheduler reserves for a container when choosing a node. A limit is the hard ceiling the container may not exceed while running. One is about placement, the other about enforcement.

If requests are too low compared to real usage, the scheduler overcommits a node and Pods fight for resources the node does not have. If limits are too low, the container is throttled or killed even though the node has spare capacity.

They do not have to match. When limits are higher than requests, the Pod is Burstable: it can use more than its request when the node has room. When requests equal limits, the Pod is Guaranteed, which gets the best protection when the node is under pressure.

Reference: Kubernetes docs - Resource management

Going over the memory limit gets the container killed immediately (OOMKilled, exit code 137). Going over the CPU limit only slows it down (throttling). The two are not treated the same way, and that trips up many debugging sessions.

Memory cannot be taken back from a running process without killing it, so the kernel's OOM killer ends the container the moment it crosses its limit, with no warning in the app's own logs. CPU is a rate, so a container at its limit is just scheduled less often. Requests get slower, but it keeps running.

So "the app looks fine in the logs but requests time out" is often a CPU limit set too low, while "the app just disappears and restarts" is often OOMKilled. kubectl describe pod shows Reason: OOMKilled and Exit Code: 137 in the last state when memory was the cause.

Reference: Kubernetes docs - Resource management

Choose a StatefulSet when each Pod needs a stable identity and its own persistent storage that survives rescheduling, such as databases or message brokers. A Deployment treats every Pod as interchangeable. A StatefulSet does not.

Each Pod gets a stable name and DNS entry (db-0, db-1, db-2) that stays the same even if it moves to another node. Each one also gets its own PersistentVolumeClaim, so db-1 always comes back with its own disk.

By default Pods are created and scaled in order, so db-0 must be Ready before db-1 starts. That matters when a primary must exist before replicas join.

The cost is complexity. StatefulSets scale more slowly and are harder to reason about during updates. Deleting one does not delete its PVCs by default, which is a safety feature but a common source of "why is old data still here".

Reference: Kubernetes docs - StatefulSets

All three influence where Pods land, but from different sides. nodeSelector and node affinity are the Pod saying "I want a node like this". Taints and tolerations are the node saying "do not schedule here unless you are allowed".

nodeSelector is the simplest: an exact label match, all or nothing. Node affinity does the same with more power. It has hard rules (requiredDuringSchedulingIgnoredDuringExecution), soft preferences (preferredDuringSchedulingIgnoredDuringExecution) and operators like In, NotIn and Exists.

A taint on a node repels every Pod that lacks a matching toleration. That is how you reserve nodes for GPUs, a dedicated team, or nodes being drained, without editing every other Pod to keep it away.

To dedicate a node pool you usually do both: taint the nodes to keep everyone else out, and add affinity to the right Pods to pull them in.

Reference: Kubernetes docs - Assigning Pods to nodes

The HPA runs a control loop. It compares observed metrics (usually CPU or memory use, optionally custom or external metrics) with a target you set, then changes the replica count of a Deployment or StatefulSet to close the gap, within the min and max you define.

For CPU or memory scaling it needs the Metrics Server installed, because the HPA does not collect metrics itself. It only reads them from the metrics API.

It does not react to every blip. It has a stabilization window and avoids scaling down hard right after scaling up, so one noisy spike does not cause churn.

It also scales replica count, not the size of each Pod. For that you would use the Vertical Pod Autoscaler. Running both on the same metric for the same workload is a known source of conflicting decisions.

Reference: Kubernetes docs - Horizontal Pod Autoscaling

A Role grants permissions inside one Namespace. A ClusterRole grants permissions across the whole cluster, or can be reused in several Namespaces through separate RoleBindings. Both list allowed verbs (get, list, create, delete) on resource types. The difference is scope, not power.

A Role bound with a RoleBinding only applies in its own Namespace. A ClusterRole bound with a ClusterRoleBinding applies everywhere. A ClusterRole bound with a RoleBinding applies only inside that RoleBinding's Namespace, which is a handy way to reuse one "view only" role across many teams.

RBAC is purely additive. There is no explicit deny, so a subject's permissions are the union of everything every binding grants. Good least-privilege design starts with nothing and adds narrowly, instead of starting from cluster-admin and trying to subtract.

Reference: Kubernetes docs - RBAC authorization

It depends on how the ConfigMap was added to the Pod. If it is mounted as a volume, the kubelet eventually syncs the new file into the container, usually within about a minute, with no restart. If it is injected as an environment variable, the value is fixed at Pod start and never changes in a running Pod. You must recreate the Pod.

Even with a volume, "the file changed" does not mean "the app noticed". Most apps read config once at startup. Unless the app watches the file, the new content sits unused until the next restart.

That is why kubectl rollout restart deployment/myapp is the practical way to make a ConfigMap change take effect. It forces a real rolling update instead of hoping every container re-reads its config.

Reference: Kubernetes docs - ConfigMaps

A PersistentVolume (PV) is a piece of real storage in the cluster, such as an EBS volume, an NFS share or a local disk. A PersistentVolumeClaim (PVC) is a request for storage made by a Pod, stating how much space and which access mode it needs. Kubernetes binds a PVC to a matching PV. The Pod only refers to the PVC.

This indirection separates "what storage exists" from "what the app asked for", much as a Service separates Pod IPs from a stable address.

With dynamic provisioning, a StorageClass creates the PV on demand when a PVC asks for it. That is how most clusters work today, so you rarely create PVs by hand.

Access modes matter too: ReadWriteOnce (one node at a time), ReadOnlyMany, and ReadWriteMany (many nodes at once, supported only by some backends such as NFS or EFS). Trying to attach an RWO volume to Pods on two nodes is a common cause of Pods stuck in Pending with a mount error.

Reference: Kubernetes docs - Persistent Volumes

A PodDisruptionBudget (PDB) sets a minimum number of Pods of an app that must stay available during a voluntary disruption, such as a node drain, a cluster autoscaler scale-down or kubectl drain. Kubernetes refuses to evict Pods if that would break the floor, until replacements are running elsewhere.

The key word is voluntary. A PDB cannot stop involuntary events like a node crash or an OOM kill. What it stops is routine maintenance taking down every replica at once.

You set either a minimum available count or a maximum unavailable count. For a 3-replica service, minAvailable: 2 stops a drain from evicting two Pods at the same time.

Reference: Kubernetes docs - Disruption budgets

A Job runs one or more Pods to completion, once, and tracks success. A Deployment keeps Pods alive forever, a Job does not. A CronJob is a Job on a schedule, written in standard cron syntax. It creates a new Job at each scheduled time.

Use a Job for one-off work such as a database migration or a batch export. It retries failed Pods up to backoffLimit and then gives up.

CronJobs add scheduling gotchas. concurrencyPolicy decides whether a new run may start while the last one is still going (Allow, Forbid or Replace). startingDeadlineSeconds decides how late a missed run may still start. Both matter when overlapping runs would corrupt data, or when a backlog of missed runs should not all fire at once after an outage.

Reference: Kubernetes docs - CronJob

An init container runs to completion before any regular container in the Pod starts. If there are several, they run in order. The Pod does not move on until each one exits successfully. Regular containers start together and are expected to keep running.

That ordering guarantee is the value. Use init containers for setup the app should not do itself: waiting for a dependency, running a one-time migration, cloning a config repo into a shared volume, or fixing permissions on a mounted volume.

If an init container fails, the Pod stays stuck, shown as Init:CrashLoopBackOff or Init:Error. To diagnose it, read the init container's logs with kubectl logs <pod> -c <init-container-name>, not the main container's.

Reference: Kubernetes docs - Init containers

A NetworkPolicy limits which Pods can talk to which others, and on which ports. It is enforced by the cluster's network plugin (CNI). Without any NetworkPolicy, the default is fully open: any Pod can reach any other Pod on any port.

Once a policy selects a Pod, that Pod becomes default-deny in the directions the policy covers (ingress, egress or both). Only traffic a rule allows gets through. Everything else is dropped silently.

The classic mistake: a team applies default-deny egress for security and forgets to allow DNS (port 53). Every service lookup then fails, and it looks like a random app problem instead of a network block.

Also remember that not every CNI enforces NetworkPolicy. On plugins that do not, the object does nothing. Confirm what your cluster runs before relying on it for isolation.

Reference: Kubernetes docs - Network Policies

Both bring external HTTP(S) traffic into the cluster. Ingress is one fairly small resource. The Gateway API is a family of resources that split routing into roles and can express much more without vendor-specific extras.

Ingress only standardizes basic host and path routing. Anything more (traffic splitting, rewrites, timeouts, canaries) usually goes through controller-specific annotations, so the same YAML can behave differently on NGINX, Traefik or HAProxy.

The Gateway API puts those behaviours into typed, portable fields. It also separates concerns: a GatewayClass and Gateway (owned by the platform team, defining the listener and load balancer) and routes such as HTTPRoute or GRPCRoute (owned by app teams) that attach to it. It also supports gRPC, TCP and UDP routing as first-class features.

The Gateway API is a Kubernetes SIG Network project and new clusters increasingly start with it. Some popular Ingress controllers have announced retirement or reduced support, so check the current status of the controller you use before planning around it.

Reference: Kubernetes docs - Gateway API

An admission controller intercepts a request to the API server (create, update or delete) after authentication and authorization, but before the object is saved to etcd. It can change or reject the request. This is where cluster-wide policy is enforced beyond what RBAC can express.

Mutating webhooks run first and can modify the object: inject a sidecar, set default resource limits, add a required cost label. Validating webhooks run after all mutations and can only accept or reject the final object, for rules like "every Deployment must have resource limits" or "no Pod may run as root".

Both call an HTTP service you run, or a policy engine such as OPA Gatekeeper or Kyverno that handles the plumbing. If that service is slow or down, the webhook's failurePolicy decides whether requests are rejected or let through. A broken webhook set to Fail can block every deployment in the cluster until it is fixed.

Reference: Kubernetes docs - Dynamic admission control

Pod Security Standards define three levels of limits on what a Pod may do in security-sensitive areas, such as running as root, using host networking or host paths, adding Linux capabilities or running privileged containers.

• privileged allows almost anything.
• baseline blocks the most dangerous settings while staying broadly compatible.
• restricted enforces hardened practice: non-root, no privilege escalation, dropped capabilities and a seccomp profile.

They are enforced by the built-in Pod Security Admission controller, configured per Namespace with labels such as pod-security.kubernetes.io/enforce: restricted. It replaced PodSecurityPolicy, which was removed because its rules were hard to reason about.

It has three modes: enforce (reject), audit (log) and warn (show a warning). A safe rollout is to set the target level to warn first, watch what would break, and switch to enforce once nothing unexpected shows up.

Reference: Kubernetes docs - Pod Security Standards

The HPA scales out: it changes how many replicas run. The VPA scales up: it changes how much CPU and memory each Pod's containers request. They solve different problems and can fight each other if used on the same metric of the same workload.

HPA fits workloads that replicate easily, like a stateless API. VPA fits workloads that cannot be replicated easily, or when requests were simply sized wrong. A single-instance component that needs more memory is not helped by a second replica. It needs a bigger container.

VPA in Auto mode evicts and recreates Pods with new values, which causes a short disruption each time. In Off (recommendation-only) mode it just tells you what it would set, which is a good way to right-size requests without letting it touch running Pods.

Running HPA on CPU and VPA on the same container's CPU is a documented anti-pattern. VPA changing the request changes what "100% utilization" means, and the two can scale in opposite directions.

Reference: Kubernetes docs - Autoscaling workloads

Topology spread constraints tell the scheduler to spread Pods evenly across a domain such as zones, racks or nodes. Anti-affinity is a yes or no rule ("not on the same node as that Pod"). Spread constraints are about balance, and let you say how much imbalance is acceptable.

The key field is maxSkew, the maximum difference in Pod count between the fullest and emptiest domain. With maxSkew: 1 across 3 zones and 6 replicas, no zone can have more than one extra Pod compared to the emptiest zone.

whenUnsatisfiable decides what happens if balance cannot be met. DoNotSchedule is a hard rule (a Pod may stay Pending). ScheduleAnyway is a soft preference.

Anti-affinity becomes awkward with many replicas and zones, because pairwise rules do not express "evenly across 3 zones" well. A spread constraint says it in one rule, so it is usually the preferred tool for "do not put all replicas in one failure domain".

Reference: Kubernetes docs - Topology spread constraints

Both are ways kube-proxy implements Service routing. The difference interviewers care about is scale. iptables mode evaluates Service rules as a long list, so lookup cost grows with the number of Services. IPVS mode uses an in-kernel hash table for choosing backends, so lookups stay fast even with thousands of Services.

IPVS still depends on iptables for some cases, such as source IP rewriting, so it does not replace it completely.

Be careful not to give a stale answer. A newer nftables mode now exists as the modern Linux alternative to iptables, IPVS mode has been marked for deprecation in recent releases, and some CNI plugins such as Cilium skip kube-proxy rules entirely with eBPF. Check the release notes for your Kubernetes version.

The safe framing: know why IPVS was created (iptables did not scale well), and know the ecosystem has kept moving since.

Reference: Kubernetes docs - kube-proxy

A native sidecar is a container listed under initContainers with restartPolicy: Always. It starts before the main containers like an init container, but it keeps running for the whole life of the Pod instead of exiting.

Before this existed, sidecars were a second entry in containers, which had two problems. There was no guarantee the sidecar (say, a mesh proxy) was ready before the app started sending traffic through it. And on shutdown, Kubernetes could kill it before the app had finished its last requests.

A native sidecar fixes both. Kubernetes waits for it to be ready before starting the main containers, and stops it only after the main containers have exited. It also helps Jobs: a Job with a sidecar no longer stays "running" forever just because the sidecar never exits.

Reference: Kubernetes docs - Sidecar containers

A normal ClusterIP Service gives one virtual IP that load-balances across ready Pods. You talk to "the Service", never to a specific Pod. A headless Service (clusterIP: None) has no virtual IP. DNS returns the individual Pod IPs, so each StatefulSet Pod can be reached by its own stable name.

This matters because clustered stateful apps often need a specific member, not any replica. A database replica must reach the primary, not whichever Pod a load balancer picks.

With a headless Service, db-0.db-service.namespace.svc.cluster.local resolves to exactly Pod db-0, and the name survives rescheduling. That is the real mechanism behind a StatefulSet's "stable network identity". A ClusterIP Service cannot do this, because hiding individual replicas from the caller is exactly what it is for.

Reference: Kubernetes docs - StatefulSets, stable network ID

A ResourceQuota caps the total resources a whole Namespace can use: the sum of all Pods' requests and limits, plus object counts such as the number of Pods, Services or PVCs. A LimitRange sets defaults and bounds for each individual object in a Namespace, such as the minimum and maximum a single container may request, and what value to use if the Pod spec gives none.

They are usually used together because of a sharp edge. Once a Namespace has a CPU or memory quota, every new Pod there must specify requests and limits for that resource, or it is rejected. A LimitRange supplies sensible defaults so teams do not suddenly get deployments blocked when a quota appears.

In short: LimitRange keeps any one Pod from being absurdly large or completely unsized. ResourceQuota keeps the Namespace as a whole from taking more than its fair share.

Reference: Kubernetes docs - Resource quotas

RollingUpdate (the default) replaces old Pods with new ones gradually, controlled by maxSurge (extra Pods allowed above the desired count) and maxUnavailable (Pods allowed to be missing). The goal is zero downtime, at the cost of old and new versions running side by side for a while. Recreate stops every old Pod first, then starts the new ones, so there is a gap with zero Pods running.

Recreate sounds worse, but it is correct when two versions must never run together. Examples: a schema-breaking migration where old and new code cannot share the data, or a ReadWriteOnce volume that only one Pod can mount, where a rolling update would stall because the new Pod cannot mount it until the old one lets go.

The trap is picking Recreate "to be safe" and accepting downtime on every deploy, when the problem it solves usually does not apply.

Reference: Kubernetes docs - Deployment strategy

Every Pod gets a Quality of Service class based on its requests and limits:

When a node runs low on resources, the kubelet reclaims them by evicting Pods. BestEffort Pods are first in line. Next come Burstable Pods that are using more than they requested. Guaranteed Pods are the last to go. The kernel's OOM killer follows a similar order through oom_score_adj values.

Practically, this is why production workloads should always set requests, and critical ones should sit in the Guaranteed class. A Pod with no requests is treated as the least important thing on the node.

Reference: Kubernetes docs - Pod QoS classes

A PriorityClass gives Pods a priority number. When a high-priority Pod cannot be scheduled because the cluster is full, the scheduler can preempt: it evicts lower-priority Pods from a node to make room, then places the important Pod there.

Node-pressure eviction is different. It is the kubelet protecting a node that is running out of memory, disk or similar, and it picks victims by QoS and usage compared to requests (priority is considered too). Preemption is the scheduler making space for a specific Pending Pod.

Use PriorityClasses to make sure critical system or customer-facing workloads win over batch jobs. Kubernetes ships system-cluster-critical and system-node-critical for core components.

The trap is giving everything a high priority. Priorities only mean something relative to each other, and too many high-priority Pods lead to constant preemption churn.

Reference: Kubernetes docs - Pod priority and preemption

Both manage Kubernetes manifests across environments, but with opposite ideas. Helm uses templates: you write YAML with placeholders and fill them from values. Kustomize uses overlays: you keep plain, valid YAML as a base, then layer patches on top for each environment, with no templating language.

Helm is stronger when you distribute software to others (a chart with a clear set of options), and when you want release tracking with upgrade and rollback. Kustomize is stronger when you own the manifests and only need small per-environment differences, like replica counts or image tags. It is built into kubectl (kubectl apply -k).

Their weaknesses mirror each other. Helm templates can become hard to read once they have many conditionals. Kustomize patches can get fiddly for large changes.

They are not exclusive. A common pattern is to render a third-party Helm chart and then patch the result with Kustomize.

Reference: Kubernetes docs - Kustomize

When you install a chart, Helm renders the templates with your values, sends the result to the API server, and records a release with a revision number. Each helm upgrade creates a new revision, and helm rollback re-applies an older revision's manifests as a new revision.

helm template renders the YAML locally and prints it, without contacting the cluster. helm install --dry-run (or helm upgrade --dry-run) renders it and also checks it against the cluster, but installs nothing. Both are how you review what will change before it does.

The common trap: Helm only knows about what it deployed. If someone edits a live object by hand with kubectl, a later helm upgrade may overwrite or conflict with that change. Also, helm rollback does not roll back data or database migrations, only the Kubernetes manifests.

The --atomic flag rolls back automatically if the upgrade fails.

Reference: Helm docs - Helm commands

GitOps means a Git repository is the single source of truth for what runs in the cluster, and an agent inside the cluster (Argo CD or Flux) continuously pulls from Git and makes the cluster match it. To change production you change Git, usually through a pull request.

In a classic push-based pipeline, CI runs kubectl apply or helm upgrade and needs credentials to the cluster. In GitOps the cluster pulls, so those credentials never leave the cluster, and the agent keeps checking for drift. If someone edits a live object by hand, the agent can detect it and revert it (self-heal).

Other benefits: a full audit trail in Git history, easy rollback with git revert, and the same process for every environment.

The trade-offs: you need a clear repo layout, a way to handle secrets safely (sealed or external secrets, never plain text in Git), and a plan for ordering and dependencies between apps.

Reference: Argo CD docs - Overview

The Cluster Autoscaler adds and removes nodes. It watches for Pods that are Pending because no node has room. When it finds them, it simulates which node group (such as an AWS Auto Scaling group) could fit them and asks the cloud to grow that group. It also removes nodes that stay underused for a while, if their Pods can be moved elsewhere.

It works with node groups you define in advance, each with one machine type and min and max sizes. Karpenter (popular on AWS) skips fixed groups. It looks at the pending Pods' requirements and launches whichever instance type fits best, and it also consolidates nodes to save cost.

Remember the relationship with HPA: the HPA adds Pods, and the autoscaler adds nodes when those Pods have nowhere to go. They work as a chain.

Scale-down respects PodDisruptionBudgets and will not remove a node with Pods that cannot be rescheduled, such as ones using local storage, unless told otherwise.

Reference: Kubernetes docs - Node autoscaling

The Container Network Interface (CNI) plugin sets up networking for Pods: it gives each Pod an IP address and connects it to the network when the kubelet creates it. Common plugins are Calico, Cilium, Flannel and cloud-specific ones such as the AWS VPC CNI.

Kubernetes expects a simple network model. Every Pod gets its own IP, every Pod can reach every other Pod without NAT, and nodes can reach Pods too. How that is achieved (overlay network with VXLAN, direct routing, cloud virtual network) is the plugin's job.

Choosing a CNI is a real decision. Some enforce NetworkPolicy and some do not. Some use eBPF for speed and observability, and some replace kube-proxy. If the CNI breaks, new Pods stay in ContainerCreating with a network setup error, and cross-node traffic can fail while same-node traffic still works.

Reference: Kubernetes docs - Cluster networking

A service mesh adds a layer that handles service-to-service traffic so apps do not each have to. Typical features: automatic mutual TLS between services, retries and timeouts, traffic splitting for canaries, and detailed metrics and tracing. Istio and Linkerd are well-known examples.

Traditionally this works by injecting a proxy sidecar (such as Envoy) next to every app container. Newer designs, such as Istio's ambient mode, move that work to per-node proxies to cut overhead.

It is worth it when you have many services and need consistent security and traffic control without changing app code, for example a requirement that all internal traffic be encrypted.

It is not free. A mesh adds latency, memory use, upgrade work and one more thing to debug. For a handful of services, an Ingress, NetworkPolicies and good libraries are often enough. Saying "it depends on scale and needs" is the right interview answer.

Reference: Kubernetes docs - Services, load balancing and networking

A Service has a label selector. A controller watches Pods that match it and records the IPs of the ready ones in EndpointSlice objects. kube-proxy (or the CNI) reads those slices and programs routing so traffic to the Service goes to one of those addresses.

EndpointSlices replaced the older single Endpoints object. One Endpoints object held every Pod IP for a Service, so in big Services a small change meant resending a huge object to every node. EndpointSlices split the list into small chunks, which scales much better. The old Endpoints API is deprecated in recent versions.

Only Pods that pass their readiness probe are included. That is how a rolling update avoids sending traffic to a Pod that is not ready yet.

When debugging, kubectl get endpointslices -l kubernetes.io/service-name=my-svc is the quickest way to see if a Service has any backends at all.

Reference: Kubernetes docs - EndpointSlices

CoreDNS gives every Service a DNS name in the form <service>.<namespace>.svc.cluster.local. Inside the same Namespace you can use just <service>. From another Namespace use <service>.<namespace>. A headless Service returns the individual Pod IPs instead of one virtual IP.

This short-name behaviour comes from the Pod's /etc/resolv.conf, which lists search domains and sets ndots:5. If a name has fewer than 5 dots, the resolver tries each search domain first before the name as written.

That means a lookup for an external name like api.example.com (2 dots) can trigger several failed internal queries before it succeeds. At scale this adds latency and load on CoreDNS. Common fixes are writing fully qualified names with a trailing dot (api.example.com.), lowering ndots in the Pod's dnsConfig, or running NodeLocal DNSCache.

Reference: Kubernetes docs - DNS for Services and Pods

A Pod uses its ServiceAccount's token. The kubelet mounts a token file into the Pod, and client libraries send it with each request. The API server validates it and then checks RBAC for that ServiceAccount.

Modern clusters use bound, projected tokens. They are short-lived, tied to the specific Pod (they stop working when the Pod is deleted), and can carry an audience so they are only valid for one purpose. The kubelet refreshes them automatically. This is much safer than the old style, where a long-lived token was stored in a Secret that never expired.

Since Kubernetes 1.24, those long-lived Secrets are no longer created automatically for ServiceAccounts. If you really need a long-lived token for something outside the cluster, you must create it explicitly, and you should treat it like a password.

Reference: Kubernetes docs - Service accounts

kubectl cordon marks a node unschedulable. No new Pods land on it, but the Pods already there keep running. kubectl drain cordons the node and then evicts its Pods so they are recreated elsewhere, respecting PodDisruptionBudgets. kubectl uncordon makes the node schedulable again.

Use cordon when you want to stop new work but not disturb existing work. Use drain before maintenance, an upgrade or removing a node.

Drain has sharp edges. DaemonSet Pods cannot be moved, so you add --ignore-daemonsets. Pods using emptyDir lose their data, which needs --delete-emptydir-data as an explicit acknowledgement. Bare Pods with no controller will not come back, so drain refuses unless you use --force.

If a PDB cannot be satisfied, drain waits instead of breaking it, which is the behaviour you want.

Reference: Kubernetes docs - Safely drain a node

A taint has a key, a value and an effect. The effect says what happens to Pods that do not tolerate it.

• NoSchedule: new Pods without a matching toleration are not scheduled on the node. Pods already running stay.
• PreferNoSchedule: a soft version. The scheduler tries to avoid the node but may use it if it has to.
• NoExecute: new Pods are blocked and existing Pods without a toleration are evicted.

NoExecute can be paired with tolerationSeconds, which lets a Pod stay for a set time before eviction. Kubernetes uses this itself: when a node becomes unreachable, it adds a NoExecute taint, and Pods tolerate it for 300 seconds by default before being evicted.

The trap: tainting a node with NoExecute carelessly can evict important Pods right away, including ones you forgot about.

Reference: Kubernetes docs - Taints and tolerations

A static Pod is managed directly by the kubelet on one node, from a manifest file in a watched folder (often /etc/kubernetes/manifests), without the API server or any controller deciding it should exist. The kubelet starts it, restarts it if it dies, and creates a read-only mirror Pod in the API so you can see it with kubectl.

kubeadm uses this to start the control plane. The API server, scheduler, controller manager and etcd run as static Pods on the control plane nodes. This solves a chicken-and-egg problem: you cannot ask the API server to start the API server.

Practical consequences: you cannot delete a static Pod with kubectl delete (the mirror comes back), and to change one you edit the file on the node. When a control plane component misbehaves, checking the manifest file and the kubelet logs is often the fastest route.

Reference: Kubernetes docs - Static Pods

When you delete a Pod, it enters Terminating and two things start at the same time:

1. The Pod is removed from Service endpoints, so new traffic stops being routed to it.
2. The kubelet runs the preStop hook (if any), then sends SIGTERM to the main process.

The container then has terminationGracePeriodSeconds (30 seconds by default) to finish. If it is still running when time is up, the kubelet sends SIGKILL.

Because endpoint removal and SIGTERM happen in parallel, a few requests can still arrive after the app starts shutting down, since the update has to spread to every node. A common fix is a short preStop sleep so the app keeps serving while routing catches up. The app should also handle SIGTERM by finishing in-flight requests and then exiting.

The trap is shells. If the container's main process is a shell script that does not forward signals, the app never sees SIGTERM and is killed hard after the grace period.

Reference: Kubernetes docs - Pod termination

ImagePullBackOff means the kubelet could not pull the image and is waiting longer between retries. ErrImagePull is the first failed attempt. Check kubectl describe pod for the exact error. The usual causes:

• The image name or tag is wrong, or the tag was never pushed.
• It is a private registry and the Pod has no (or wrong) imagePullSecrets.
• The node cannot reach the registry (network, firewall, DNS).
• The registry is rate-limiting you, which is common with public registries.

imagePullPolicy controls when the kubelet pulls. Always checks the registry every time the Pod starts. IfNotPresent uses a cached copy if the node has one. Never only uses local images. If you do not set it, it defaults to Always for the latest tag or no tag, and IfNotPresent otherwise.

That is why using latest is discouraged. You cannot tell which version is running, and rollbacks are unreliable. Pin a specific tag or, better, an image digest.

Reference: Kubernetes docs - Images

A plain Deployment only does rolling updates, so these strategies need a bit more.

Blue-green: run two full Deployments, blue (current) and green (new), then switch a Service's selector from one to the other. Rollback is switching it back. The cost is running double capacity during the switch.

Canary: send a small share of traffic to the new version first. The simplest way is two Deployments behind one Service with different replica counts (1 canary Pod next to 9 stable ones gives roughly 10 percent). It is crude because the split follows Pod count. For precise weights you use an Ingress controller with weighted routing, a service mesh, or the Gateway API.

Tools such as Argo Rollouts add progressive delivery on top: step-by-step traffic increases, automatic metric checks and automatic rollback.

The trap is forgetting the data layer. Two app versions running together share one database, so schema changes must work with both versions, usually by making them backward compatible.

Reference: Kubernetes docs - Deployments

An Ingress can reference a Secret of type kubernetes.io/tls holding a certificate and private key in its tls section. The Ingress controller then terminates HTTPS for those hosts using that certificate.

Managing those certificates by hand means remembering to renew them. cert-manager automates it. You define an Issuer or ClusterIssuer (for example Let's Encrypt through the ACME protocol) and add an annotation to the Ingress, or create a Certificate resource. cert-manager requests the certificate, proves domain ownership, stores it in the Secret, and renews it before it expires.

The two usual ACME challenges are HTTP-01 (proves ownership through a path on port 80) and DNS-01 (adds a DNS record, and is required for wildcard certificates).

Common failure: the challenge cannot complete because DNS does not point at the cluster yet, or a firewall blocks port 80. kubectl describe certificate and the challenge resources show where it got stuck.

Reference: cert-manager docs - Securing Ingress resources

Owner references link a child object to its parent. A Pod owned by a ReplicaSet, which is owned by a Deployment, forms a chain. When you delete the parent, the garbage collector deletes the children automatically. You can choose background deletion (parent goes first, children cleaned up afterwards), foreground deletion (children first), or orphaning (children are kept).

Finalizers are keys on an object that block its final removal. When you delete such an object, Kubernetes sets a deletion timestamp but keeps it until the controller that owns the finalizer finishes its cleanup, such as deleting a cloud load balancer or disk, and removes the key.

That is why an object can get stuck in Terminating: the controller that should clear the finalizer is gone, broken, or failing. You can remove a finalizer manually, but that skips the cleanup and can leave orphaned cloud resources, so it is a last resort.

Reference: Kubernetes docs - Finalizers

They answer different questions and often run together.

• metrics-server gives a short, in-memory view of current CPU and memory use for nodes and Pods. It powers kubectl top and the HPA. It keeps no history.
• Prometheus scrapes metrics from many sources on a schedule and stores them as time series, so you can graph trends, alert on them and look back at an incident.
• kube-state-metrics turns the state of Kubernetes objects into metrics, such as "how many replicas are desired versus available" or "which Pods are in CrashLoopBackOff". It reads the API, not the containers.

Roughly, metrics-server is for autoscaling and quick checks, Prometheus is for monitoring and alerting, and kube-state-metrics tells you what the cluster thinks is happening.

The trap is expecting kubectl top to show history or to work without metrics-server installed. It fails with an error if the metrics API is not available.

Reference: Kubernetes docs - Resource metrics pipeline

Containers should write logs to stdout and stderr. The container runtime captures them and the kubelet stores them on the node (under /var/log/pods), which is what kubectl logs reads. The files are rotated, and they disappear when the Pod is deleted or the node is lost.

Kubernetes has no built-in long-term log storage. The standard solution is a log agent, such as Fluent Bit or Vector, running as a DaemonSet on every node. It reads the log files and ships them to a central system like Elasticsearch, Loki or a cloud logging service.

A sidecar is the alternative when an app writes to files instead of stdout. The sidecar tails the file from a shared volume and forwards it.

Good practice: log in a structured format such as JSON, include a request or trace ID, and never log secrets. If kubectl logs shows nothing, remember it only sees stdout and stderr, not log files written inside the container.

Reference: Kubernetes docs - Logging architecture

The reclaimPolicy decides what happens to the underlying volume when its claim is deleted.

• Delete: the PV and the real disk are deleted too. This is the default for dynamically provisioned volumes, so deleting the PVC usually destroys the data.
• Retain: the PV and data are kept, but the PV moves to Released and is not reused until an admin cleans it up.

This catches people out. Deleting a namespace or a Helm release can delete PVCs and, with Delete, the data with them. For important data, use a StorageClass with Retain, take snapshots or backups (for example with CSI volume snapshots or Velero), and protect the PVCs.

Storage drivers today follow the Container Storage Interface (CSI). If the StorageClass has allowVolumeExpansion: true, you can grow a volume by editing the PVC's requested size. Shrinking is not supported.

Reference: Kubernetes docs - Persistent Volumes, reclaiming

Pending means the Pod was accepted but is not running yet. The cause is nearly always in the events at the bottom of kubectl describe pod, usually a FailedScheduling message.

Common reasons:

• Not enough resources. No node has enough free CPU or memory for the Pod's requests ("Insufficient cpu").
• Taints. Every node has a taint the Pod does not tolerate.
• Selectors and affinity. A nodeSelector, node affinity or required anti-affinity rule matches no node.
• Storage. The PVC is unbound or no storage class can provide a volume, or the volume is in a different zone.
• Quota. A ResourceQuota in the Namespace blocks creation (this usually fails earlier, at admission).

The message names the count of nodes for each reason, such as "0/5 nodes are available: 3 Insufficient memory, 2 node(s) had untolerated taint". Read that line carefully, it is the answer.

A different symptom is ContainerCreating, which means the Pod is already on a node and the problem is image pulls, volumes mounts or networking.

Reference: Kubernetes docs - Debug Pods

The scheduler runs a cycle for each unscheduled Pod. A Pod waits in a scheduling queue (ordered by priority), then goes through:

1. Filtering. Remove nodes that cannot run the Pod: not enough requested resources, untolerated taints, failed selectors, volume or port conflicts.
2. Scoring. Rank the remaining nodes with scoring plugins. Examples: prefer nodes with more free resources, nodes that already have the image, or nodes that improve spread.
3. Binding. Pick the highest score (ties are broken randomly) and write the node name to the Pod. The kubelet takes over from there.

If nothing passes filtering, the scheduler may try preemption, and if that does not work the Pod stays Pending and is retried when the cluster changes.

The scheduler decides placement using requests, not real usage, which is why wrongly sized requests lead to bad packing.

Reference: Kubernetes docs - Scheduling framework

CrashLoopBackOff is a status, not a root cause. It means the container keeps exiting and the kubelet waits longer between retries (10s, 20s, 40s, up to 5 minutes). Empty logs usually mean the container dies before the app writes anything, so log-first debugging will not work. Start elsewhere.

1. Run kubectl describe pod <name> and read Events first. OOMKilled, a failed probe or an image problem shows up here before the app ever logs.
2. Run kubectl logs <name> --previous. This reads the previous crashed instance. The current attempt may have no output yet, but the last one often printed something on its way down.
3. Check the exit code in "Last State". 137 means SIGKILL, almost always OOMKilled or a manual kill. 1 or another app code means the process exited by itself, which points to app logic or config.
4. Rule out a bad liveness probe. A healthy but slow-starting app killed by an impatient probe looks exactly like a crash. Temporarily remove the probe, or check initialDelaySeconds.
5. Attach an ephemeral debug container with kubectl debug to get a shell in the Pod's namespaces without changing it. This is essential for distroless images that have no shell.

The trap: trying to kubectl exec into a Pod that is actively crash-looping. It is a moving target. Reproduce it in a throwaway copy with the entrypoint overridden to sleep infinity, then investigate.

Reference: Kubernetes docs - Debug running Pods

The 4th replica stays Pending forever, and the other three keep running normally. A requiredDuringSchedulingIgnoredDuringExecution anti-affinity rule is a hard constraint. The scheduler will not put two matching Pods on one node however much capacity it has, and with 3 nodes there is no legal place left.

kubectl describe pod on the pending replica shows an event like "0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules". The Deployment shows 3/4 ready but does not fail or roll back.

It stays that way until a 4th node joins, or you relax the rule to preferredDuringSchedulingIgnoredDuringExecution, which the scheduler tries to follow but will not block on.

The lesson: required rules must be sized against real cluster capacity, not just written to express intent. The cluster autoscaler can add a node in response to the pending Pod, but only if it is configured and your cloud account has room to grow.

Reference: Kubernetes docs - Inter-pod affinity and anti-affinity

A stalled rolling update almost always means the new Pods are not passing their readiness probe. The Deployment will not remove more old Pods or add more new ones beyond maxUnavailable and maxSurge until the new Pods it already created are Ready. The controller is behaving correctly, it is just blocked.

Run kubectl get pods -l app=<name> to find the new-revision Pods that are not Ready, then kubectl describe pod on one. Common causes, roughly by likelihood:

• The new image really fails its readiness probe (bad build, missing config, a new dependency).
• A ConfigMap or Secret the new revision needs does not exist yet.
• A resource request that fits no node, leaving Pods Pending instead of unready.
• An image pull failure on a private registry (ImagePullBackOff).

kubectl rollout status keeps saying "waiting" unless you set progressDeadlineSeconds, so a stuck rollout can sit there silently with no alert.

Once you confirm it is broken and not just slow, the fastest recovery is usually kubectl rollout undo to return to the last good revision, then fix the new one offline instead of debugging against production traffic.

Reference: Kubernetes docs - Rolling update

Base64 is an encoding, not encryption. There is no key, and decoding is a one-line command. By default, anyone who can read a Secret through the API (subject to RBAC), or who can read the etcd data files or a snapshot, can recover every Secret in plain text. "It is a Secret object" is organization, not confidentiality, unless you add real controls.

The hardening comes in layers:

• Encryption at rest. Configure an EncryptionConfiguration on the API server so Secret data is encrypted before it is written to etcd. Without it, an etcd backup is just as readable as the live API, which is easy to forget when backups get a different security review.
• Tight RBAC. Grant get, list and watch on secrets per Namespace to only the identities that need them. Broad view roles can include Secrets by accident. RBAC is additive, so one overly broad ClusterRoleBinding undoes every narrow Role.
• Restrict etcd access. Use TLS for client and peer traffic, no direct network exposure, and treat snapshots as sensitive as the live cluster.
• External secret managers such as Vault or cloud KMS-backed stores for the most sensitive values, with Kubernetes holding only a short-lived reference.

Reference: Kubernetes docs - Encrypting data at rest

The goal is to stop the Pod cleanly and start its replacement elsewhere with the same PVC reattached, never having two Pods write to the same volume, and without skipping the app's own shutdown. "Just delete the Pod" is dangerous here.

1. Confirm a PodDisruptionBudget exists before draining, so the drain cannot take more replicas than the app tolerates. For a 3-member quorum system that is likely minAvailable: 2.
2. Give the app a real preStop hook and grace period. For anything with in-memory state or connection pools, an abrupt SIGKILL after too short a grace period risks a partial write. A preStop hook can flush buffers, leave the peer group and finish in-flight writes first.
3. Run kubectl drain <node> --ignore-daemonsets. It cordons the node (nothing new schedules there), then evicts Pods while respecting the PDB. It waits instead of forcing past the budget.
4. Check the storage backend supports clean detach and reattach. For ReadWriteOnce cloud block volumes, Kubernetes handles detach from the old node and attach to the new one. If a PVC is stuck Terminating afterwards, a leftover finalizer is usually the cause. Remove it by hand only after confirming no active I/O, because forcing it while a mount is live is how you get corruption.
5. Let the StatefulSet do its job. The replacement comes back with the same ordinal and the same PVC by design. Do not manually delete PVCs or reorder things.

Reference: Kubernetes docs - Safely drain a node

Flapping usually means the HPA is reacting to noise in a metric, not to a real, sustained change in load. It is a metrics and tuning problem, not proof that autoscaling is broken.

Common causes:

• Badly sized requests. The HPA targets a percentage of the request, so a request that is too low or too high makes utilization swing wildly for the same real load.
• Bursty metrics. A periodic batch job, a cache warm-up or a noisy neighbour can push the metric over the target for a few seconds at a time.
• Aggressive behaviour settings. The HPA has a stabilization window to prevent this, but if it has been shortened or the scaling policies allow big steps, small wobbles become replica churn.
• Slow readiness. New Pods take time to become Ready. If the metric does not account for not-yet-ready Pods well, the HPA sees load as still high and scales up again before the new Pods absorb any traffic.

The fix is almost never "stop autoscaling". Right-size the requests first, widen the scale-down stabilization window (the safe direction to be cautious in), and consider scaling on a steadier signal such as requests per second through a custom metric if CPU is naturally bursty for this workload.

Reference: Kubernetes docs - HPA scaling behavior

Use an ephemeral debug container: a temporary container that Kubernetes attaches to a running Pod's existing namespaces (network, process and optionally filesystem), without restarting the Pod and without the original image needing any tools. It is the purpose-built answer for distroless and scratch images.

--target=app shares the process namespace of the named container. From inside the debug container, ps aux shows the real app process, /proc/<pid>/ lets you inspect open files, and curl localhost:8080 reaches the app exactly as if you had a shell in it.

Two related tools solve different problems:

• kubectl debug --copy-to makes a full copy of the Pod with a changed spec, useful for changing the entrypoint or limits to reproduce a problem without touching the live Pod.
• kubectl debug node/<name> starts a debug container in the host's namespaces, for node-level problems such as a full disk or a bad kernel setting that you cannot see from inside any Pod.

Reference: Kubernetes docs - Ephemeral debug container

No single mechanism gives real multi-tenancy. It is a combination, each part covering a different failure. The common wrong answer is "give each team a Namespace" and stop there.

• Namespaces as the organizing boundary, one per team or environment. On their own they do not stop cross-namespace traffic, limit resources or restrict kernel-level behaviour. They are the scope everything else attaches to.
• RBAC per Namespace, so a team can manage its own workloads but cannot read or change another team's objects or Secrets. Least privilege by default.
• ResourceQuotas and LimitRanges so one team cannot use the whole cluster. A quota without default requests and limits still lets one bad Pod ask for unreasonable amounts, so apply both.
• NetworkPolicies, starting from default-deny per Namespace with explicit allows. This is what actually stops one team's Pods from reaching another's.
• Pod Security Admission at restricted (or at least baseline), so workloads cannot run as root, use host networking or escalate privileges and affect the shared node.
• Separate clusters for truly hostile or strictly regulated tenants. Namespace isolation inside one cluster is often judged insufficient there, however well configured. Name that ceiling instead of implying config always solves it.

Reference: Kubernetes docs - Multi-tenancy

NotReady means the control plane has not received a recent heartbeat from that node's kubelet. It does not mean the Pods have stopped. The API server just has stale information. The Pods might be running fine on a node whose kubelet cannot report in, or they might already be dead and Kubernetes has not noticed. The debugging path must allow for both.

First, run kubectl describe node <name> and read the condition and reason. Causes include the kubelet being down, the container runtime crashing, disk or memory pressure, or a network partition between the node and the control plane. With node access, check systemctl status kubelet and journalctl -u kubelet, and the same for the runtime.

Timing is built in. After the node controller stops hearing from the node (about 40 seconds by default), it marks the node condition Unknown and adds the not-ready or unreachable taint. Pods tolerate that taint for 300 seconds by default (tolerationSeconds), then they are evicted and controllers create replacements elsewhere.

The catch: the control plane cannot stop the originals on an unreachable node. For a while the same Pod can appear to run in two places, until the node recovers and the kubelet reconciles, or someone confirms the node is gone and deletes it.

The trap is assuming NotReady means the workload is down and force-killing Pods or forcing a failover before confirming whether the node is unreachable or truly dead. The right action differs, and guessing wrong risks split-brain.

Reference: Kubernetes docs - Node status

Cluster DNS failures are rarely "DNS is broken". They are usually "something about these Pods' path to CoreDNS is broken". The fact that only some Pods fail is the most useful clue. Compare what the failing and working Pods share and where they differ:

• Same node or different nodes. If all failing Pods are on one node, or it recently recovered from NotReady, suspect stale conntrack entries for UDP DNS. The kernel can keep a mapping to a CoreDNS Pod that no longer exists, so new queries go nowhere until the entry expires or is cleared.
• CoreDNS health. Run kubectl get pods -n kube-system -l k8s-app=kube-dns. CoreDNS can be Running but not Ready, for example when it cannot reach the Kubernetes API. Its logs then loop on still waiting on: "kubernetes", and it is quietly removed from the endpoints.
• NetworkPolicy. A default-deny egress policy on only some Namespaces, or added recently, that forgot to allow port 53 to kube-system breaks DNS for exactly the Pods it covers. Check this early.
• Resolver config on the Pod. Run kubectl exec and read /etc/resolv.conf. A custom dnsPolicy or dnsConfig on one workload can point it at the wrong nameserver.
• A direct test. Run kubectl run -it --rm dnsutils --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- nslookup kubernetes.default from both a working and a failing Namespace or node. It tells you whether the problem is node-level, Namespace-level or Pod-spec-level.

Reference: Kubernetes docs - Debugging DNS resolution

The core idea is that the control plane and the workloads are largely decoupled. Kubelets keep running Pods even if they briefly cannot reach the API server, and a careful rollout touches only one component or node at a time.

1. Read the release notes and API deprecations first. Kubernetes supports upgrading one minor version at a time (1.30 to 1.31, not straight to 1.32). Manifests that use an API version removed in the target release must be migrated before upgrading, or they will fail to apply afterwards.
2. Upgrade the control plane first, one component at a time: kube-apiserver, then the controller manager and scheduler, then etcd if it needs a new version. On managed platforms (EKS, GKE, AKS) this is done for you, and the impact is usually a brief API unavailability, not a workload outage.
3. Upgrade kubelets node by node with normal drain discipline: cordon, respect PodDisruptionBudgets, wait for Pods to reschedule, upgrade, uncordon. PDBs defined ahead of time matter most here.
4. Roll node pools instead of upgrading in place where the platform allows. Create new nodes on the target version, drain old ones gradually, then delete them. No kubelet is ever mid-upgrade in an odd state.
5. Validate before you call it done. Check for Pods stuck Pending from a capacity gap during the rollout, and for any workload still using a removed API version that slipped through step 1.

The trap: treating "the control plane is highly available" as "workloads will not notice". An HA control plane prevents an outage, but a careless kubelet rollout that ignores PDBs can still take down more replicas than intended.

Reference: Kubernetes docs - Upgrading kubeadm clusters

The risk is a window where two processes both believe they own the same volume: the old container, if it is still running on a node the API server lost contact with, and the new Pod that the force-delete told Kubernetes to treat as already gone.

A normal kubectl delete pod waits for the kubelet to confirm the container has stopped. --grace-period=0 --force skips that and just removes the object from the API. That is why it is a forced action and not the default.

"Unresponsive" does not mean "stopped". If the node is only cut off from the control plane, the process can still be alive with the volume mounted. Force-deleting the Pod object does not kill it. It only frees the identity for a replacement. If the new Pod is scheduled on another node and mounts the same volume, you get two live writers with no coordination: a classic split-brain, and one of the worst ways to corrupt data in Kubernetes.

The safer sequence:

1. Confirm whether the node is unreachable or truly dead. Use kubectl describe node and direct node access if possible, not just Pod status.
2. If the node is confirmed dead (hardware failure, terminated instance) and will not return, force-delete is legitimate, because no process is left to conflict.
3. If the node might still be alive, fence it first. Cordon it, and if you have out-of-band access (cloud API, IPMI, hypervisor), power it off or confirm the process has stopped before force-deleting the Pod object.
4. Prefer the built-in timeout path when there is no urgency. The default grace period before eviction exists precisely so nobody jumps to force-delete without confirmation.

Reference: Kubernetes docs - Force delete StatefulSet Pods

Work from the Service backwards towards the Pod, one layer at a time.

1. Does the Service have endpoints? Run kubectl get endpointslices -l kubernetes.io/service-name=<svc>. If the list is empty, the selector matches no ready Pods. Compare kubectl get svc <svc> -o yaml selector with kubectl get pods --show-labels. A typo here is the most common cause.
2. Are the Pods Ready? A Pod that fails its readiness probe is removed from the endpoints. kubectl get pods shows 0/1 in READY.
3. Is the port right? The Service's targetPort must be the port the app actually listens on. Test from inside the Pod: kubectl exec <pod> -- curl localhost:<port>. If that fails, the problem is the app, not Kubernetes. Also check the app listens on 0.0.0.0 and not only 127.0.0.1.
4. Does DNS resolve? From a debug Pod, run nslookup <svc>. If DNS fails, see the cluster DNS question.
5. Can you reach the Pod IP directly, and the ClusterIP? Reaching the Pod IP but not the ClusterIP points at kube-proxy or the CNI. Check kube-proxy Pods and logs.
6. Is a NetworkPolicy blocking it? A default-deny policy without a matching allow rule drops traffic silently.

Testing from a throwaway debug Pod in the same Namespace separates "the Service is broken" from "my client is broken".

Reference: Kubernetes docs - Debug Services

Eviction is the kubelet protecting a node that is running out of a resource such as memory, disk or inodes. It is not the same as an OOM kill, where the kernel kills a container that went over its own memory limit.

The kubelet compares signals such as memory.available or nodefs.available to thresholds. A hard threshold (for example memory.available below 100Mi by default) triggers eviction immediately. A soft threshold must hold for a grace period first. The node gets a condition like MemoryPressure or DiskPressure, and new Pods stop being scheduled there.

To pick victims, the kubelet considers whether a Pod's usage exceeds its requests, then its priority, then how far over it is. In practice that means BestEffort and over-request Burstable Pods go first. Evicted Pods show status Evicted and stay as records until cleaned up.

To investigate:

1. kubectl describe node for pressure conditions and events.
2. kubectl get pods -A --field-selector=status.phase=Failed and describe a victim to read the eviction message.
3. Check what is using the resource, for example logs filling the disk or a Pod with no limits using too much memory.
4. Fix the root cause: set realistic requests and limits, add node capacity, or cap ephemeral storage use.

Raising eviction thresholds only hides the problem.

Reference: Kubernetes docs - Node-pressure eviction

etcd holds all cluster state, so a snapshot is your recovery point if the control plane is lost. On a kubeadm cluster you take one with etcdctl, passing the endpoint and the TLS certificates:

To restore, you stop the control plane, restore the snapshot into a new data directory (with etcdutl snapshot restore in recent versions), point the etcd static Pod manifest at that directory, and let the control plane start again. In a multi-member cluster, all members must be restored consistently from the same snapshot.

A snapshot alone is not a full backup:

• It holds objects, not the data inside PersistentVolumes. Use volume snapshots or a tool such as Velero for that.
• It does not hold the certificates in /etc/kubernetes/pki or any external things like cloud load balancers.
• A snapshot contains Secrets, so it must be encrypted and access-controlled.

The most important rule: practise the restore. A backup you have never restored is only a hope. On managed Kubernetes you usually cannot touch etcd, so you rely on provider features and tools like Velero or GitOps to rebuild.

Reference: Kubernetes docs - Operating etcd clusters

The impact differs a lot by component, and the key idea is that running workloads are largely independent of the control plane.

The practical lesson: a control plane outage is serious, but it does not instantly take your apps down. It does remove self-healing, so a second failure during the outage (a node crash, say) is not repaired.

That is why production control planes run 3 or 5 replicas across zones, and why etcd quorum (a majority) matters. With 3 members you can lose 1, with 5 you can lose 2.

Reference: Kubernetes docs - Cluster architecture

The most likely cause is expired certificates. In a kubeadm cluster the component certificates (API server, etcd, front-proxy and so on) are valid for one year by default, and the cluster CA for ten. If no one upgraded the cluster or renewed them in that time, they expire and components stop trusting each other. Often the API server stops working and kubectl fails with "x509: certificate has expired".

To check, run kubeadm certs check-expiration on a control plane node. To fix:

1. Renew with kubeadm certs renew all on each control plane node.
2. Restart the control plane static Pods so they load the new certificates. Move their manifests out of /etc/kubernetes/manifests and back, or restart the kubelet.
3. Update your admin kubeconfig, because admin.conf was among the renewed certificates, and copy it to where you use it.

Kubelet client certificates are normally rotated automatically, but the control plane ones are not unless you upgrade (a kubeadm upgrade renews them).

The prevention is simple and boring: monitor certificate expiry with an alert well in advance, and keep clusters upgraded. Managed services handle this for you.

Reference: Kubernetes docs - Certificate management with kubeadm

This is almost always the race in Pod termination. When an old Pod is deleted, the endpoint removal and the SIGTERM happen at the same time. Routers (kube-proxy, the Ingress controller, cloud load balancers) learn about the removal a moment later. During that gap they still send requests to a Pod that is already shutting down, and if the app exits at once, those requests fail.

Other contributors: new Pods marked Ready before they can really serve (a weak readiness probe), keep-alive connections that stay attached to old Pods, and an Ingress controller that reloads its config slowly.

Fixes, usually combined:

1. Add a preStop sleep (about 5 to 15 seconds) so the Pod keeps serving while routing converges.
2. Handle SIGTERM properly. Stop accepting new connections, finish in-flight requests, then exit. Make sure terminationGracePeriodSeconds is longer than the sleep plus the longest request.
3. Use a real readiness probe that only passes when the app can serve traffic.
4. Set maxUnavailable: 0 with a maxSurge, so a new Pod is Ready before an old one is removed.
5. Keep a PodDisruptionBudget so drains and autoscaling do not remove too many Pods at once.

If the errors continue, check the load balancer's own connection draining and idle timeout settings, because they can be the real source.

Reference: Kubernetes docs - Pod termination

Think in layers, and say what failure each layer survives.

• Control plane. Run 3 control plane nodes (or 5 for larger setups) across different availability zones, with a load balancer in front of the API servers. etcd needs a majority to work, so an odd number matters. You can use stacked etcd (on the control plane nodes, simpler) or external etcd (more isolation, more to run). A managed service removes most of this work.
• Worker nodes. Spread them across zones in several node groups. Use topology spread constraints so replicas of one app do not all land in the same zone, and keep spare capacity so a zone loss can be absorbed.
• Workload settings. At least 2 or 3 replicas for important services, PodDisruptionBudgets, readiness probes, sensible requests and limits, and graceful shutdown.
• Storage. Zonal disks do not follow a Pod to another zone. Use replicated storage, application-level replication or regional disks for stateful systems.
• Operations. Tested etcd and volume backups, monitoring and alerting, a planned upgrade process, and infrastructure as code or GitOps so the cluster can be rebuilt.
• Beyond one cluster. A single cluster survives a zone failure but not a region failure or a bad control plane upgrade. For that, run a second cluster in another region and decide how traffic fails over and how data is replicated.

The key interview point is to match the design to the real requirement. More redundancy costs money and complexity, so ask what outage you must survive.

Reference: Kubernetes docs - Production environment

The disk the PVC is bound to lives in one availability zone, and the scheduler is trying to place the Pod on a node in a different zone. Block volumes such as AWS EBS or GCP persistent disks are zonal, so the Pod can only run in the zone where the disk exists.

This often happens because the volume was created before the Pod was scheduled. With the default volumeBindingMode: Immediate, the volume is provisioned as soon as the PVC is created, in whatever zone the provisioner picks, with no knowledge of where the Pod will fit. The Pod then may need a node that has no room in that zone, or its other rules (affinity, spread constraints) push it to another zone.

The standard fix for future volumes is a StorageClass with volumeBindingMode: WaitForFirstConsumer. The volume is only created after the scheduler picks a node, so the disk lands in the same zone as the Pod.

For an existing stuck volume you have a few options: add capacity in that zone (the cluster autoscaler needs a node group in that zone), relax the Pod's conflicting rules, or restore the data from a snapshot into the zone you want.

Reference: Kubernetes docs - Volume binding mode

The cause can be CPU throttling. CPU limits are enforced by the Linux CFS scheduler in short periods, 100 ms by default. A container gets a quota of CPU time per period (a 500m limit means 50 ms of CPU time in each 100 ms). Once it has used its quota, it is paused until the next period.

An app with many threads can use its whole quota in a few milliseconds and then sit frozen for the rest of the period, even though the average over a minute looks tiny. Requests that land in a throttled window see added latency, often in short, repeating spikes. Average CPU graphs hide this completely.

To confirm, look at the throttling metrics, for example container_cpu_cfs_throttled_periods_total compared with container_cpu_cfs_periods_total. A high ratio of throttled periods means the limit is hurting you.

Options:

• Raise the CPU limit, or reduce the app's thread count to fit it.
• Remove the CPU limit but keep an accurate CPU request. The request still protects the Pod's fair share under contention, and many teams do this for latency-sensitive services. It is a trade-off, because a runaway Pod can then use idle CPU on its node.
• Use the Guaranteed class with whole-CPU values for the most sensitive workloads.

The key point: requests and limits serve different purposes, and a limit set too tight can hurt even when the node is mostly idle.

Reference: Kubernetes docs - Resource management

The autoscaler only acts when it can prove that adding a node from some node group would let the Pending Pod run. If it cannot, it does nothing. Check:

1. Is the Pod actually unschedulable for a capacity reason? The autoscaler ignores Pods that are Pending for reasons a new node will not fix, such as an unbound PVC, a bad selector or a wrong scheduler name.
2. Would the Pod fit on a new node at all? If its requests are larger than the biggest node type in any group, no scale-up helps.
3. Do the node group's labels and taints match? If the Pod needs a label, a GPU or a zone that no node group provides, the autoscaler cannot satisfy it.
4. Is the group already at its maximum size, or blocked by a cloud quota or a lack of capacity for that instance type in that zone?
5. Zone and storage constraints. A Pod tied to a zonal volume needs a node group that can add nodes in exactly that zone.
6. Is the autoscaler healthy? Read its logs and its status ConfigMap. They state why each Pod was skipped, for example "pod didn't trigger scale-up: 3 max node group size reached".
7. Missing requests. Pods without resource requests give the autoscaler poor information.

Start with the logs. They almost always name the exact reason.

Reference: Kubernetes docs - Node autoscaling

Start with direct questions. kubectl auth can-i tells you what a subject can do, and with --as you can test another identity:

For a wider view, list ClusterRoleBindings and RoleBindings, and look for cluster-admin bindings, wildcards (* on verbs or resources) and groups like system:authenticated that were granted too much. Tools such as kubectl-who-can and rbac-tool help answer "who can read Secrets?". API audit logs show what was actually used.

Patterns that quietly give more power than they look:

• create pods in a Namespace is nearly the same as reading its Secrets, because the Pod can mount any Secret or ServiceAccount there. It can also run a privileged container if Pod Security does not block it.
• get, list or watch on secrets exposes every secret value in scope.
• The escalate and bind verbs on roles let someone grant permissions they do not hold themselves.
• impersonate lets a subject act as another user or group.
• Editing Roles or RoleBindings, or creating a ServiceAccount token for a powerful account.
• exec into Pods gives access to whatever those Pods can do.

Fix by starting from least privilege, avoiding wildcards, using per-app ServiceAccounts, reviewing bindings regularly and combining RBAC with Pod Security and admission policies.

Reference: Kubernetes docs - RBAC good practices

That pattern narrows it down a lot. Same-node traffic stays on the node's local bridge, and cross-node traffic needs the network plugin's path between nodes. So the fault is in how the CNI moves packets between nodes, not in the app, the Service or DNS.

Check in this order:

1. Confirm the pattern with Pod IPs. Use two debug Pods on different nodes and curl or ping the Pod IP directly. This removes Services and kube-proxy from the picture.
2. Check the CNI Pods (a DaemonSet in kube-system or its own Namespace) on both nodes. A crashed or unready CNI Pod on one node breaks that node's cross-node traffic. Read its logs.
3. Look at the underlay network. Overlay modes need ports open between nodes: VXLAN uses UDP 8472 for Flannel (Calico uses 4789 by default), and IP-in-IP uses protocol 4. Cloud security groups, firewalls or a recent network change very often block these.
4. Check MTU. Overlays add headers, so a wrong MTU makes small packets (pings, DNS) work while large ones (TLS, big responses) hang. This looks like random, size-dependent failures.
5. Check routes and node networking. Look at the node's routing table for each Pod CIDR, and for IP forwarding or reverse path filtering settings that drop packets.
6. Rule out NetworkPolicy by testing in a Namespace without policies, or by checking the CNI's policy logs.

The most common real causes are blocked overlay ports and MTU mismatches.

Reference: Kubernetes docs - Cluster networking

Deleting a Namespace deletes everything inside it first. The Namespace controller must list every resource type, delete the objects and wait for their finalizers to clear. If any step cannot finish, the Namespace stays Terminating.

The usual causes:

• An object with a finalizer that never completes, because its controller is gone. For example, a custom resource whose operator was uninstalled before the Namespace was deleted.
• An unavailable aggregated API. If an APIService (such as the metrics API) is down, the controller cannot discover all resource types, and the Namespace reports a discovery failure.
• A PVC or Pod that cannot be removed, for example a volume that will not detach.

To diagnose, read the Namespace's status conditions, which usually name the problem:

The first command shows the conditions, and the second lists what is still left in the Namespace.

Resolve it at the cause. Fix or restore the missing controller or the broken APIService, so the finalizers clear normally. Only if that is impossible should you remove the stuck object's finalizer by hand.

The shortcut of editing the Namespace's finalize field to force it away is dangerous. It can leave orphaned cloud resources such as load balancers and disks, and hides the underlying problem. Treat it as a last resort, after you have checked what will be left behind.

Reference: Kubernetes docs - Namespaces