ImagePullBackOff

Kubernetes hub

A Kubernetes pod status indicating the kubelet cannot pull the container image from the registry and is applying exponential backoff delays between retry attempts. It is always caused by one of four things: the image name or tag is wrong, the image does not exist, the registry requires authentication that the cluster does not have, or the registry is unreachable from the node.

ImagePullBackOff — Why Your Pod Can't Start

What Is ImagePullBackOff in Simple Terms?

Before a pod can run, Kubernetes must pull the container image onto the node. ImagePullBackOff means that pull is failing — and Kubernetes is retrying with increasing wait times (10s, 20s, 40s... up to 5 minutes) rather than hammering the registry endlessly.

◈ DIAGRAM

+------------------------------------------+
| Pod scheduled on mumbai-worker-1         | <- Step 1: Scheduler picks a node
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet tries to pull the image          | <- Step 2: Contacts the registry
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Pull fails                               | <- Step 3: Registry rejects or unreachable
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Status: ErrImagePull                     | <- Step 4: First failure label
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet retries with backoff timer       | <- Step 5: Retry at 10s, 20s, 40s...
| Status: ImagePullBackOff                 |
+------------------------------------------+

Remember
ErrImagePull and ImagePullBackOff are the same root problem. ErrImagePull is the first attempt, ImagePullBackOff is what you see once Kubernetes starts spacing out retries. Always fix the underlying cause — retries will not resolve it on their own.

The Four Root Causes — In Order of Frequency

◈ DIAGRAM

+------------------------------------------+
| Cause 1: Wrong image name or tag         | <- Most common. Typo in deployment YAML
+------------------------------------------+
| Cause 2: Image does not exist in registry| <- Tag was deleted or never pushed
+------------------------------------------+
| Cause 3: Missing registry credentials   | <- Private registry, no imagePullSecret
+------------------------------------------+
| Cause 4: Registry unreachable from node | <- Network issue, VPC firewall, DNS
+------------------------------------------+

How to Diagnose — Step by Step

Bash

# Step 1 — Read the exact error message from pod events
kubectl describe pod <pod-name> -n production
 
# Look for this section at the bottom of the output:
# Events:
#   Type     Reason     Message
#   ----     ------     -------
#   Warning  Failed     Failed to pull image "registry.razorpay.in/api:v2.5":
#                       rpc error: code = Unknown desc = pulling image:
#                       unauthorized: authentication required
 
# Step 2 — Confirm what image name is in the spec
kubectl get pod <pod-name> -n production \
  -o jsonpath='{.spec.containers[*].image}'
# Output: registry.razorpay.in/api:v2.5

Fix 1 — Wrong Image Name or Tag

The most common cause. A tag that does not exist on the registry returns a manifest unknown error.

YAML

# deployment.yaml — verify these three parts exactly
spec:
  containers:
    - name: api
      image: registry.razorpay.in/api:v2.5.1   # registry / name / tag
      #       ^^^^^^^^^^^^^^^^^  ^^^  ^^^^^
      #       registry host      repo  tag — all three must be exact

Bash

# Verify the tag exists in the registry before deploying
# For Docker Hub:
docker pull nginx:1.25-alpine
 
# For a private ECR registry:
aws ecr describe-images \
  --repository-name api \
  --region ap-south-1 \
  --query 'imageDetails[*].imageTags'

Fix 2 — Private Registry Needs Authentication

When pulling from a private registry (ECR, GCR, self-hosted Harbor), the node needs credentials. These are stored as a Kubernetes Secret of type kubernetes.io/dockerconfigjson and referenced in the pod spec as imagePullSecrets.

Bash

# Create the registry credential Secret
kubectl create secret docker-registry ecr-credentials \
  --docker-server=905418385260.dkr.ecr.ap-south-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region ap-south-1) \
  -n production

YAML

# deployment.yaml — reference the Secret in the pod spec
spec:
  imagePullSecrets:
    - name: ecr-credentials      # Must exist in the same namespace as the pod
  containers:
    - name: api
      image: 905418385260.dkr.ecr.ap-south-1.amazonaws.com/api:v2.5.1

Security
ECR tokens expire every 12 hours. For production clusters at Razorpay or Zerodha, use the amazon-ecr-credential-helper or an IRSA-based solution instead of a static Secret — static tokens cause ImagePullBackOff every 12 hours when the token silently expires.

Fix 3 — Registry Unreachable from Node

If the image name and credentials are correct but the pull still fails, the node cannot reach the registry. This is common when private registries are inside a VPC and worker nodes are in a different subnet.

Bash

# SSH into the affected node and test registry connectivity directly
ssh rahul@10.0.1.50
 
# Test TCP connectivity to the registry
nc -zv registry.razorpay.in 443
# Expected: Connection to registry.razorpay.in 443 port [tcp/https] succeeded!
 
# Test DNS resolution of the registry hostname
nslookup registry.razorpay.in
# If this fails, check your VPC DNS settings and /etc/resolv.conf on the node
 
# Attempt a manual pull from the node
crictl pull registry.razorpay.in/api:v2.5.1
# The error output here is far more detailed than what kubectl shows

Quick Troubleshooting Reference

Error Message	Root Cause	Fix
`manifest unknown`	Tag does not exist in the registry	Push the correct tag or fix the tag name in the deployment
`unauthorized: authentication required`	Missing or expired credentials	Create or refresh `imagePullSecrets`
`pull access denied`	Image is private, wrong credentials	Verify username and password in the docker-registry Secret
`no such host`	Registry DNS not resolving on the node	Check node `/etc/resolv.conf` and VPC DNS configuration
`connection refused`	Registry unreachable on port 443	Check firewall rules and security group for the node
`context deadline exceeded`	Network timeout to registry	Check VPC route tables, NAT gateway, or proxy config

Bash

# Full diagnostic sequence — run these in order
# 1. Read the exact error
kubectl describe pod <pod-name> -n production | grep -A 10 Events
 
# 2. Confirm the image string in the spec
kubectl get pod <pod-name> -n production -o jsonpath='{.spec.containers[*].image}'
 
# 3. Check if imagePullSecret exists in the right namespace
kubectl get secret ecr-credentials -n production
 
# 4. Test the secret content is valid
kubectl get secret ecr-credentials -n production \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq
 
# 5. Force a fresh pull attempt by deleting and recreating the pod
kubectl delete pod <pod-name> -n production
# The deployment controller will recreate it immediately

Common Mistake
Creating the imagePullSecrets Secret in the default namespace but deploying the pod to production. Secrets are namespace-scoped — the Secret must exist in the same namespace as the pod that references it. A Secret in default is completely invisible to pods in production.

Tip
On EKS clusters, attach the AmazonEC2ContainerRegistryReadOnly IAM policy to your node group's IAM role. Worker nodes will authenticate to ECR automatically without any imagePullSecrets — ECR credentials are handled transparently via instance metadata. This is the cleanest solution for Hotstar or Swiggy scale where dozens of services pull from the same ECR registry.

Frequently Asked Questions

Why does Kubernetes retry with exponential backoff instead of failing fast on a bad image pull?

Backoff exists because pull failures are often transient — a registry rate-limiting requests, a brief network partition, or a credential rotation still propagating — so kubelet retries with increasing delay (starting around 10 seconds, capping around 5 minutes) rather than hammering the registry or giving up permanently. ImagePullBackOff is the waiting state between retries; the actual failure reason is always in the preceding ErrImagePull event.

What's the fastest way to diagnose which of the four causes is behind an ImagePullBackOff?

Run `kubectl describe pod <name>` and read the Events section — it states the exact registry error (unauthorized, manifest not found, name unknown, connection timed out). A common mistake is assuming it's always a typo and re-editing the manifest repeatedly; just as often it's a missing `imagePullSecrets` reference for a private registry, or the node's outbound network/DNS being unable to reach the registry at all, which `describe pod` will show directly.