What is ImagePullBackOff? | DevOps Dictionary

ImagePullBackOff — Why Your Pod Can't Start

What Is ImagePullBackOff in Simple Terms?

Before a pod can run, Kubernetes must pull the container image onto the node. ImagePullBackOff means that pull is failing — and Kubernetes is retrying with increasing wait times (10s, 20s, 40s... up to 5 minutes) rather than hammering the registry endlessly.

◈ DIAGRAM

+------------------------------------------+
| Pod scheduled on mumbai-worker-1         | <- Step 1: Scheduler picks a node
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet tries to pull the image          | <- Step 2: Contacts the registry
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Pull fails                               | <- Step 3: Registry rejects or unreachable
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Status: ErrImagePull                     | <- Step 4: First failure label
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet retries with backoff timer       | <- Step 5: Retry at 10s, 20s, 40s...
| Status: ImagePullBackOff                 |
+------------------------------------------+

📌 Remember: ErrImagePull and ImagePullBackOff are the same root problem. ErrImagePull is the first attempt, ImagePullBackOff is what you see once Kubernetes starts spacing out retries. Always fix the underlying cause — retries will not resolve it on their own.

The Four Root Causes — In Order of Frequency

◈ DIAGRAM

+------------------------------------------+
| Cause 1: Wrong image name or tag         | <- Most common. Typo in deployment YAML
+------------------------------------------+
| Cause 2: Image does not exist in registry| <- Tag was deleted or never pushed
+------------------------------------------+
| Cause 3: Missing registry credentials   | <- Private registry, no imagePullSecret
+------------------------------------------+
| Cause 4: Registry unreachable from node | <- Network issue, VPC firewall, DNS
+------------------------------------------+

How to Diagnose — Step by Step

Bash

1# Step 1 — Read the exact error message from pod events
2kubectl describe pod <pod-name> -n production
3 
4# Look for this section at the bottom of the output:
5# Events:
6#   Type     Reason     Message
7#   ----     ------     -------
8#   Warning  Failed     Failed to pull image "registry.razorpay.in/api:v2.5":
9#                       rpc error: code = Unknown desc = pulling image:
10#                       unauthorized: authentication required
11 
12# Step 2 — Confirm what image name is in the spec
13kubectl get pod <pod-name> -n production \
14  -o jsonpath='{.spec.containers[*].image}'
15# Output: registry.razorpay.in/api:v2.5

Fix 1 — Wrong Image Name or Tag

The most common cause. A tag that does not exist on the registry returns a manifest unknown error.

YAML

1# deployment.yaml — verify these three parts exactly
2spec:
3  containers:
4    - name: api
5      image: registry.razorpay.in/api:v2.5.1   # registry / name / tag
6      #       ^^^^^^^^^^^^^^^^^  ^^^  ^^^^^
7      #       registry host      repo  tag — all three must be exact

Bash

1# Verify the tag exists in the registry before deploying
2# For Docker Hub:
3docker pull nginx:1.25-alpine
4 
5# For a private ECR registry:
6aws ecr describe-images \
7  --repository-name api \
8  --region ap-south-1 \
9  --query 'imageDetails[*].imageTags'

Fix 2 — Private Registry Needs Authentication

When pulling from a private registry (ECR, GCR, self-hosted Harbor), the node needs credentials. These are stored as a Kubernetes Secret of type kubernetes.io/dockerconfigjson and referenced in the pod spec as imagePullSecrets.

Bash

1# Create the registry credential Secret
2kubectl create secret docker-registry ecr-credentials \
3  --docker-server=905418385260.dkr.ecr.ap-south-1.amazonaws.com \
4  --docker-username=AWS \
5  --docker-password=$(aws ecr get-login-password --region ap-south-1) \
6  -n production

YAML

1# deployment.yaml — reference the Secret in the pod spec
2spec:
3  imagePullSecrets:
4    - name: ecr-credentials      # Must exist in the same namespace as the pod
5  containers:
6    - name: api
7      image: 905418385260.dkr.ecr.ap-south-1.amazonaws.com/api:v2.5.1

⚠️ Security: ECR tokens expire every 12 hours. For production clusters at Razorpay or Zerodha, use the amazon-ecr-credential-helper or an IRSA-based solution instead of a static Secret — static tokens cause ImagePullBackOff every 12 hours when the token silently expires.

Fix 3 — Registry Unreachable from Node

If the image name and credentials are correct but the pull still fails, the node cannot reach the registry. This is common when private registries are inside a VPC and worker nodes are in a different subnet.

Bash

1# SSH into the affected node and test registry connectivity directly
2ssh rahul@10.0.1.50
3 
4# Test TCP connectivity to the registry
5nc -zv registry.razorpay.in 443
6# Expected: Connection to registry.razorpay.in 443 port [tcp/https] succeeded!
7 
8# Test DNS resolution of the registry hostname
9nslookup registry.razorpay.in
10# If this fails, check your VPC DNS settings and /etc/resolv.conf on the node
11 
12# Attempt a manual pull from the node
13crictl pull registry.razorpay.in/api:v2.5.1
14# The error output here is far more detailed than what kubectl shows

Quick Troubleshooting Reference

Error Message	Root Cause	Fix
`manifest unknown`	Tag does not exist in the registry	Push the correct tag or fix the tag name in the deployment
`unauthorized: authentication required`	Missing or expired credentials	Create or refresh `imagePullSecrets`
`pull access denied`	Image is private, wrong credentials	Verify username and password in the docker-registry Secret
`no such host`	Registry DNS not resolving on the node	Check node `/etc/resolv.conf` and VPC DNS configuration
`connection refused`	Registry unreachable on port 443	Check firewall rules and security group for the node
`context deadline exceeded`	Network timeout to registry	Check VPC route tables, NAT gateway, or proxy config

Bash

1# Full diagnostic sequence — run these in order
2# 1. Read the exact error
3kubectl describe pod <pod-name> -n production | grep -A 10 Events
4 
5# 2. Confirm the image string in the spec
6kubectl get pod <pod-name> -n production -o jsonpath='{.spec.containers[*].image}'
7 
8# 3. Check if imagePullSecret exists in the right namespace
9kubectl get secret ecr-credentials -n production
10 
11# 4. Test the secret content is valid
12kubectl get secret ecr-credentials -n production \
13  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq
14 
15# 5. Force a fresh pull attempt by deleting and recreating the pod
16kubectl delete pod <pod-name> -n production
17# The deployment controller will recreate it immediately

🔴 Common Mistake: Creating the imagePullSecrets Secret in the default namespace but deploying the pod to production. Secrets are namespace-scoped — the Secret must exist in the same namespace as the pod that references it. A Secret in default is completely invisible to pods in production.

💡 Tip: On EKS clusters, attach the AmazonEC2ContainerRegistryReadOnly IAM policy to your node group's IAM role. Worker nodes will authenticate to ECR automatically without any imagePullSecrets — ECR credentials are handled transparently via instance metadata. This is the cleanest solution for Hotstar or Swiggy scale where dozens of services pull from the same ECR registry. ===

Syncing Data

ImagePullBackOff

Technical Explanation & Usage

ImagePullBackOff — Why Your Pod Can't Start

What Is ImagePullBackOff in Simple Terms?

The Four Root Causes — In Order of Frequency

How to Diagnose — Step by Step

Fix 1 — Wrong Image Name or Tag

Fix 2 — Private Registry Needs Authentication

Fix 3 — Registry Unreachable from Node

Quick Troubleshooting Reference

Related Terms

Namespace

Pod

Deployment