Troubleshooting ImagePullBackOff and Registry Authentication Issues

Overview and What You Will Learn

ImagePullBackOff is one of the most common errors engineers encounter when deploying to Kubernetes — and one of the most frustrating, because the same image that pulls fine on your laptop refuses to pull on the cluster. The error can be caused by five completely different root causes that all produce the same status message. This lab walks through every cause systematically with concrete diagnostic commands and fixes.

By the end of this guide you will be able to:

• Distinguish between ErrImagePull (first attempt) and ImagePullBackOff (retry with backoff) and why both map to the same root causes
• Diagnose authentication failures with private registries and create the correct imagePullSecret
• Fix image name, tag, and digest errors that cause pull failures
• Configure registry credentials for AWS ECR, GCP Artifact Registry, and private Harbor instances
• Resolve network-level pull failures caused by firewall rules or registry outages

Why This Matters in Production

At Razorpay, a CI/CD pipeline successfully built and pushed a new payments service image to their private ECR registry — but the Kubernetes deployment sat in ImagePullBackOff for 11 minutes before an engineer noticed. The root cause: the imagePullSecret referencing the ECR credentials had expired. The new pods could not pull the image, but the old pods were still running — so no alerts fired. The fix took 30 seconds once diagnosed, but discovery took 11 minutes of confusion.

At Hotstar, a developer accidentally deployed with image: video-encoder:latest instead of image: registry.hotstar.com/video-encoder:v3.2.1 — pulling from Docker Hub (which doesn't have the image) instead of their private registry. Same error message, completely different cause, completely different fix.

Core Principles

The five root causes of ImagePullBackOff — always check in this order:

CAUSE 1 — Wrong image name or tag

CAUSE 2 — Missing or incorrect imagePullSecret

Private registry requires credentials. No imagePullSecret = 401 Unauthorized from registry. Wrong secret = 403 Forbidden or "incorrect username or password"

CAUSE 3 — Expired credentials (ECR tokens expire every 12 hours)

AWS ECR tokens are time-limited. A secret created yesterday may be expired today. Symptom: worked before, suddenly fails on new pod scheduling.

CAUSE 4 — Network cannot reach the registry

Node firewall blocks outbound HTTPS to registry domain. Private registry behind VPN that cluster nodes cannot access. Symptom: "dial tcp: connection timed out" in pod events.

CAUSE 5 — Registry rate limiting (Docker Hub)

Docker Hub limits unauthenticated pulls to 100/6hr per IP. Shared NAT gateway = all nodes share one IP = rate limit hit quickly. Symptom: "toomanyrequests: You have reached your pull rate limit"

Detailed Step-by-Step Practical Lab

Step 1 — Identify the Exact Error

The pod status shows ImagePullBackOff:

ALWAYS describe the pod first — the Events section contains the actual error:

Look for the Events section at the bottom:

Remember

ErrImagePull is the first failed attempt. ImagePullBackOff is Kubernetes applying exponential backoff (10s → 20s → 40s → ... → 5min cap) before retrying. Both share the same root cause — always read the Failed to pull image line above them for the actual error message.

Step 2 — Diagnose and Fix: Wrong Image Name or Tag

Error signature: "manifest unknown" or "not found"

Check what tags actually exist in your registry. For AWS ECR:

For Harbor (private registry):

Fix: Update the deployment with the correct image reference:

Verify the image reference in the deployment spec:

Security

Never use image: myapp:latest in production manifests. The latest tag is mutable — the registry can silently replace it with a different image. Always pin to an immutable tag (v2.1.0) or image digest (sha256:abc123...) for reproducible deployments.

Step 3 — Diagnose and Fix: Missing imagePullSecret for Private Registry

Error signature: "401 Unauthorized" or "403 Forbidden"

Step 1 — Create the imagePullSecret from registry credentials.

Method A: Docker config file (most portable):

Method B: From an existing Docker config.json (if you've already logged in locally):

Verify the secret was created correctly:

Alternative: Patch an existing deployment to add imagePullSecrets:

Tip

Attach the imagePullSecret to the namespace's default ServiceAccount so every pod in the namespace automatically inherits it — eliminating the need to add imagePullSecrets to every individual deployment:

kubectl patch serviceaccount default -n production -p '{"imagePullSecrets": [{"name": "razorpay-registry-secret"}]}'

Step 4 — Diagnose and Fix: Expired AWS ECR Credentials

AWS ECR authentication tokens expire every 12 hours. A secret created during cluster setup will fail the next day:

Error signature: "no basic auth credentials" or "401" from ECR

Check when the ECR secret was last updated:

Refresh the ECR token and update the secret:

Remember

The permanent solution for ECR on EKS is to use IRSA (IAM Roles for Service Accounts) instead of static credentials. With IRSA, the node's IAM role automatically authorises ECR pulls with no secrets required — no tokens to expire, no CronJob refresh needed.

Step 5 — Diagnose and Fix: Network Cannot Reach Registry

Error signature: "connection timed out" or "no such host"

Test DNS resolution for the registry from inside a pod on the same node:

Test TCP connectivity to the registry port:

If connection times out — check node security group / firewall rules. For AWS EKS — verify the node security group allows outbound HTTPS (443) to registry IP:

For private registries — check if the registry is accessible from the VPC. Test directly from the node via SSH:

Clean up test pods:

Step 6 — Diagnose and Fix: Docker Hub Rate Limiting

Error signature: "toomanyrequests"

Check current rate limit status from inside a pod:

Fix 1: Authenticate Docker Hub pulls to get higher limits (200/6hr per account). Create a Docker Hub pull secret:

Fix 2 (Permanent): Mirror public images to your private registry. Never pull from Docker Hub directly in production — mirror images first.

Mirror a public image to your private ECR registry — pull locally, retag, push to private registry:

Update deployments to use the mirrored image:

Step 7 — Verify the Fix and Confirm Successful Pull

After applying any fix — force a new pod to attempt the pull:

Watch the new pod status:

Confirm image was pulled by checking pod events:

Verify which image digest was actually pulled:

Production Best Practices & Common Pitfalls

• Mirror all public images (Docker Hub, quay.io, gcr.io) to your private registry as part of your base image policy. Public registries have rate limits, availability incidents, and can remove images — your production cluster should never depend on them directly.
• Use image digests (image: registry.razorpay.in/payments-api@sha256:abc123...) instead of mutable tags in production GitOps manifests. Tags can be overwritten; digests are immutable.
• Rotate registry credentials on a schedule and automate the Kubernetes secret update via CI/CD or a CronJob — manual rotation always gets forgotten until a deployment fails at 2am.
• For multi-namespace clusters, attach the imagePullSecret to each namespace's default ServiceAccount rather than adding it to every deployment manifest individually. One change, universal coverage.
• Always test image pull independently of deployment configuration by running kubectl run test --image=<your-image> --restart=Never -n <ns> — this isolates the pull failure from any deployment spec issues.

Common Mistake

Deleting and recreating the pod to "force a retry" of an ImagePullBackOff. The backoff timer resets on pod recreation, but if the root cause is not fixed (wrong image name, missing secret, expired token), the new pod will fail identically. Fix the root cause first, confirmed by kubectl describe pod, before attempting any restart.

Quick Reference & Troubleshooting Commands