Troubleshooting ImagePullBackOff

Troubleshooting / Guide 89 of 103 / Beginner

• Kubernetes1.36.4
• Runtimecontainerd 2.2.6
• Cluster4 nodes
• CNICalico v3.32.1
• TimeAbout 25 min
• Reviewed21 August 2026

Written against the versions above. The pull error text comes from containerd. A different runtime words it differently while naming the same cause.

Before you start

• A cluster and kubectl.
• The Pods guide, for container status structure.
• Nothing else. This guide breaks things on purpose in a scratch namespace.

1. 1

   A tag that does not exist

   The most common form. nginx:this-tag-does-not-exist is a real repository and a fictional tag.

   After 25 seconds the status is ImagePullBackOff. Note what that word means: the kubelet has already tried, failed, and is now backing off before trying again. The interval grows to a cap of five minutes, which is why a Pod that has been failing for an hour can sit apparently idle for minutes at a time. It has not given up.

   Two statuses appear in this area and the distinction is useful:

   • ErrImagePull - a pull just failed.
   • ImagePullBackOff - it has failed at least once and is waiting before retrying.

   They alternate, so seeing either means the same thing.

   The useful field is the waiting message, and it names the cause precisely:

   failed to resolve image: docker.io/library/nginx:this-tag-does-not-exist: not found

   not found after a successful registry contact means the tag or repository is wrong. Note also that nginx was expanded to docker.io/library/nginx, which tells you which registry was actually consulted, and that is worth checking when you expected a private one.

   The events say the same thing with timing attached. Pulling image ... (x2 over 24s) and Failed ... (x2 over 22s) show two attempts, which is the retry loop visible.

   $kubectl run badimage -n tsh --image=nginx:this-tag-does-not-exist --restart=Neverpod/badimage created$sleep 25; kubectl get pod badimage -n tshNAME       READY   STATUS             RESTARTS   AGEbadimage   0/1     ImagePullBackOff   0          25s$kubectl get pod badimage -n tsh -o jsonpath="{.status.containerStatuses[0].state.waiting.reason}{\": \"}{.status.containerStatuses[0].state.waiting.message}{\"\n\"}"ImagePullBackOff: Back-off pulling image "nginx:this-tag-does-not-exist": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:this-tag-does-not-exist": failed to resolve image: docker.io/library/nginx:this-tag-does-not-exist: not found$kubectl describe pod badimage -n tsh | sed -n '/Events:/,$p' | head -12Events:  Type     Reason     Age               From               Message  ----     ------     ----              ----               -------  Normal   Scheduled  25s               default-scheduler  Successfully assigned tsh/badimage to cka1001-node03  Normal   BackOff    21s               kubelet            spec.containers{badimage}: Back-off pulling image "nginx:this-tag-does-not-exist"  Warning  Failed     21s               kubelet            spec.containers{badimage}: Error: ImagePullBackOff  Normal   Pulling    7s (x2 over 24s)  kubelet            spec.containers{badimage}: Pulling image "nginx:this-tag-does-not-exist"  Warning  Failed     5s (x2 over 22s)  kubelet            spec.containers{badimage}: Failed to pull image "nginx:this-tag-does-not-exist": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:this-tag-does-not-exist": failed to resolve image: docker.io/library/nginx:this-tag-does-not-exist: not found  Warning  Failed     5s (x2 over 22s)  kubelet            spec.containers{badimage}: Error: ErrImagePull

   Expected resultScheduled first, then the pull failures. That ordering matters: the Pod was scheduled, so this is not a scheduling problem. A Pod with an image problem has a node.

   Success conditionThe waiting message ends in not found.

2. 2

   A registry that cannot be reached, and why it looks identical

   Same symptom, different cause. registry.example.invalid/private/app:1.0 names a host that does not resolve.

   kubectl get pods shows ImagePullBackOff for both Pods. Identical. The status column cannot distinguish a typo in a tag from a network problem, expired credentials, or a registry that is down.

   The message can:

   dial tcp: lookup registry.example.invalid on 127.0.0.53:53: no such host

   That is DNS failing on the node, not in the cluster: 127.0.0.53 is systemd-resolved on the host. Node image pulls do not use CoreDNS, which is why an image pull can fail while every Pod resolves cluster names perfectly.

   So a small taxonomy of pull messages, each pointing somewhere different:

   • not found / manifest unknown - the tag or repository is wrong. Check spelling and that the tag was actually pushed.
   • no such host / dial tcp ... i/o timeout - the node cannot reach the registry. Check node DNS, egress, and any proxy configuration.
   • 401 Unauthorized / authentication required - credentials. Check imagePullSecrets on the Pod and the ServiceAccount.
   • 403 Forbidden - authenticated but not permitted. The credential is valid for the wrong repository.
   • toomanyrequests - Docker Hub rate limiting. Authenticate, or mirror the image.

   The events for this Pod show only Failed, Pulling and BackOff reasons with no detail, which is why the -o jsonpath on the waiting message is the better first command: the event list tells you *that* it failed, the message tells you *why*.

   $kubectl run noauth -n tsh --image=registry.example.invalid/private/app:1.0 --restart=Neverpod/noauth created$sleep 30; kubectl get pod noauth -n tsh -o jsonpath="{.status.containerStatuses[0].state.waiting.reason}{\": \"}{.status.containerStatuses[0].state.waiting.message}{\"\n\"}"ErrImagePull: failed to pull and unpack image "registry.example.invalid/private/app:1.0": failed to resolve image: failed to do request: Head "https://registry.example.invalid/v2/private/app/manifests/1.0": dial tcp: lookup registry.example.invalid on 127.0.0.53:53: no such host$kubectl get events -n tsh --field-selector involvedObject.name=noauth -o custom-columns=REASON:.reason,MSG:.message --no-headers | tail -3Failed      Error: ErrImagePullBackOff     Back-off pulling image "registry.example.invalid/private/app:1.0"Failed      Error: ImagePullBackOff

   Expected resultThis one caught in ErrImagePull rather than ImagePullBackOff, purely a matter of when you looked. Note the URL in the message: /v2/private/app/manifests/1.0 is the registry API path, which confirms the image reference was parsed as you intended.

   Success conditionThe message names a DNS failure rather than a missing tag.

3. 3

   Why kubectl logs cannot help you here

   The instinct is to check the logs. It cannot work, and the error explains why:

   container "badimage" in pod "badimage" is waiting to start: trying and failing to pull image

   There are no logs because the container never started. There is no container at all: the image could not be fetched, so nothing was created to produce output.

   This is worth internalising as a general rule, because it applies to a whole class of failures. kubectl logs requires a container that has run. For anything failing before that point (image pull, volume mount, admission, scheduling) the information is in kubectl describe and the Pod's status, never in logs.

   The division of labour:

   • Before the container starts - kubectl describe pod and .status.containerStatuses[].state.waiting. Image pulls, mount failures, config errors.
   • After the container starts - kubectl logs, and kubectl logs --previous for a container that has since died.

   Recognising which side of that line you are on saves the time otherwise spent asking for logs that cannot exist.

   $kubectl logs badimage -n tsh 2>&1 | tail -2Error from server (BadRequest): container "badimage" in pod "badimage" is waiting to start: trying and failing to pull image

   Expected resultA BadRequest that states the reason. The error is genuinely informative, which is unusual enough to be worth reading rather than skipping.

   Success conditionkubectl logs explains that the container is waiting to start.

Troubleshooting

• ImagePullBackOff and you do not know which of several causes it is.

  Why: The status is a category, not a diagnosis.

  Fix:One command: kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].state.waiting.message}'. Then match the text: not found is the tag, no such host or a timeout is the network, 401 is credentials, 403 is permissions, toomanyrequests is rate limiting. Do not stop at kubectl get pods.

• The image exists and you can pull it on your laptop, but the cluster cannot.

  Why: The pull happens on the node, with the node's DNS, network and credentials. None of that is your workstation's.

  Fix:Test from the node: sudo crictl pull <image> reproduces exactly what the kubelet does, including auth. It is the fastest way to separate a cluster problem from an image problem. Also check whether the node needs a proxy that your machine does not.

• 401 Unauthorized from a private registry.

  Why: No usable imagePullSecrets, or the secret is in the wrong namespace or the wrong format.

  Fix:Pull secrets are namespaced, so a secret in default does nothing for a Pod in prod. Create it with kubectl create secret docker-registry, and check the Pod actually references it: kubectl get pod <name> -o jsonpath='{.spec.imagePullSecrets}'. Attaching it to the ServiceAccount instead applies it to every Pod using that account, which is usually what you want.

• It worked yesterday and fails today with no changes.

  Why: Usually a mutable tag. :latest or :v1 can be repointed or deleted in the registry, and imagePullPolicy: Always will then fetch the new one or fail.

  Fix:Look at the message: not found on a tag that used to work means the tag moved or went. This is the argument for digests (image@sha256:...), which cannot change. Note :latest also implies imagePullPolicy: Always, so a cached copy will not save you.

• toomanyrequests from Docker Hub.

  Why: Anonymous rate limits, counted per source IP, so every node in a cluster shares a budget.

  Fix:Authenticate even for public images: an imagePullSecrets with a Docker Hub account raises the limit substantially. Better, mirror the images you depend on into your own registry so an external rate limit cannot stop a deploy.

• One node fails to pull and the others succeed.

  Why: Node-specific: DNS, proxy configuration, registry credentials in the runtime config, or a full disk.

  Fix:Compare. kubectl get pods -o wide shows which node the failures are on; if it is always the same one, the node is the variable. Check df -h (a full image filesystem produces confusing pull errors), the node's /etc/resolv.conf, and /etc/containerd/config.toml for registry settings that differ from the others.

Official sources

• Images
• Pull an image from a private registry
• Debug Pods
• Container image pull policy
• Add ImagePullSecrets to a service account