Troubleshooting CrashLoopBackOff

Troubleshooting / Guide 90 of 103 / Beginner

• Kubernetes1.36.4
• Runtimecontainerd 2.2.6
• Cluster4 nodes
• CNICalico v3.32.1
• TimeAbout 25 min
• Reviewed21 August 2026

Written against the versions above. Restart counts and ages depend on when you look. The fields to read do not.

Before you start

• The ImagePullBackOff guide, for the rule about when logs can exist.
• The probes and Pod lifecycle guide, for restart policies.
• A scratch namespace.

1. 1

   A container that fails on startup

   The container prints a line to stdout, a line to stderr, and exits 3. That is a faithful model of the real thing: a process that starts, discovers it cannot proceed, complains, and gives up.

   After 45 seconds the status reads Error, not CrashLoopBackOff, with RESTARTS 3 (30s ago).

   That is worth explaining because it confuses people who go looking for the string CrashLoopBackOff. The Pod cycles through states:

   • Error - the container just exited non-zero.
   • CrashLoopBackOff - waiting before the next attempt.
   • Running - briefly, during each doomed attempt.

   Which one kubectl get pods shows depends entirely on when you looked. The reliable signal is the restart count, and specifically that it is increasing. A Pod with 200 restarts is in a crash loop regardless of what the STATUS column says at this instant.

   The back-off is exponential: 10s, 20s, 40s, up to a five-minute cap. It resets after the container stays up for ten minutes. So a Pod that crashes every twenty minutes shows a slowly climbing restart count and never enters back-off at all.

   Notice reason= printed empty. At that moment the container was not waiting, it was between states, so state.waiting.reason was unset. The field you want is elsewhere.

   kubectl apply -f - <<'EOF'apiVersion: v1kind: Podmetadata:  name: crasher  namespace: tshspec:  containers:  - name: app    image: busybox:1.36    command: ["sh", "-c", "echo starting up; echo 'config missing, giving up' >&2; exit 3"]    resources: {requests: {cpu: 10m, memory: 16Mi}}EOFpod/crasher created$sleep 45; kubectl get pod crasher -n tshNAME      READY   STATUS   RESTARTS      AGEcrasher   0/1     Error    3 (30s ago)   45s$kubectl get pod crasher -n tsh -o jsonpath="restarts={.status.containerStatuses[0].restartCount}{\" reason=\"}{.status.containerStatuses[0].state.waiting.reason}{\"\n\"}"restarts=3 reason=

   Expected resultError with a climbing restart count. 3 (30s ago) means three restarts, most recent 30 seconds ago, which is the back-off becoming visible.

   Success conditionThe restart count is above zero and increasing.

2. 2

   lastState is where the exit code lives

   state describes the container now. lastState describes the previous incarnation, and that is the one that failed.

   Error exit=3

   The exit code is the most valuable single field in a crash loop, because the conventional meanings narrow the problem immediately:

   • 1 - a generic application error. Read the logs.
   • 2 - shell misuse, often a bad command or missing argument.
   • 126 - the command was found but is not executable.
   • 127 - command not found. Very common: a typo in command, or a binary that is not in that image.
   • 137 - SIGKILL. Almost always OOMKilled; that has its own guide.
   • 139 - SIGSEGV, a segmentation fault.
   • 143 - SIGTERM, so something asked it to stop and it did.

   Anything from 1 to 125 is the application's own choice, as the 3 here is.

   The pair to remember: 127 means your command is wrong, 137 means your memory limit is wrong. Between them they cover a large share of real crash loops, and neither needs the logs to diagnose.

   $kubectl get pod crasher -n tsh -o jsonpath="{.status.containerStatuses[0].lastState.terminated.reason}{\" exit=\"}{.status.containerStatuses[0].lastState.terminated.exitCode}{\"\n\"}"Error exit=3

   Expected resultError as the reason and the application's own exit code. Error here is a category meaning "exited non-zero"; OOMKilled or Completed would appear in the same field for those cases.

   Success conditionlastState.terminated.exitCode gives you a number to interpret.

3. 3

   The logs, and the one that got away

   kubectl logs works here, unlike the image-pull case, because the container did run. Both lines appear: stdout and stderr are interleaved into one stream, so a container's error output is not somewhere separate.

   config missing, giving up is the actual diagnosis, and in a real system this is usually where the answer is.

   Then --previous, which is the flag people are told to reach for, and here it fails:

   unable to retrieve container logs for containerd://5e45328a8...

   Worth being clear about why, because the flag is genuinely useful and this is its limitation. --previous reads the log of the *immediately preceding* container instance, and that instance's log files are removed by the runtime once it is superseded. After several rapid restarts the previous-previous logs are gone, and the runtime may have collected the previous one too.

   So the practical rules:

   • kubectl logs works whenever the current or most recent container produced output. In a fast crash loop, the current attempt is usually the useful one.
   • kubectl logs --previous is for a container that restarted once, where you want the output from before the restart. Use it immediately, not an hour later.
   • Neither survives Pod deletion. If a Pod is going to be replaced and you want its output, capture it first.

   For anything you need to keep, a log shipper collecting from the node is the real answer; kubectl logs reads files on a node with a retention policy you did not choose.

   $kubectl logs crasher -n tshstarting upconfig missing, giving up$kubectl logs crasher -n tsh --previous 2>&1 | tail -3unable to retrieve container logs for containerd://5e45328a8bcefb1bd07d55bd7f3f925f22bdc557d8b327a8b5328138820f1419

   Expected resultTwo lines from the current attempt, and a failure on the previous one. The container ID in that error is a real ID for a container that no longer has logs on disk.

   Success conditionkubectl logs shows the application's own error message.

4. 4

   The events, and a phase that lies

   Read this against step 1, because it is the same Pod nine minutes later and the status has changed by itself.

   kubectl get pod crasher now reads CrashLoopBackOff with 6 restarts. In step 1 the same Pod read Error with 3. Nothing was done to it in between. That is the alternation described earlier, caught from the other side, and it is the clearest possible argument for using the restart count rather than the status string.

   The events show the cycle plainly. Pulled, Created and Started each (x7 over 9m32s): seven full attempts, all identical, and BackOff at (x13 over 9m31s). That repeated triple is the signature of a crash loop in an event list. Seven Started events mean the container really did start seven times, so the image is fine, the mounts are fine and the scheduler is fine. The problem is inside the process.

   Note the back-off working: 7 starts in 9 minutes, and the most recent was 3m45s ago. The interval has grown to minutes, which is why a Pod failing for hours can look idle.

   Now the detail that catches people out. Look at the phase column:

   crasher    Running   CrashLoopBackOff   <none>      6

   .status.phase is Running. Not Failed, not Pending. A Pod is Running once it is bound to a node and at least one container has started, and a container in a restart loop satisfies that. This Pod will report Running forever while never working for a moment.

   The consequence is practical: do not build alerts or scripts on phase. A query for Pods that are not Running misses every crash loop in the cluster. What you want is restartCount above a threshold, or the Ready condition being false, or the READY 0/1 column, which is the honest one in kubectl get pods.

   Compare the two reason columns across the rows, too. crasher now has a waiting reason and no terminated reason, the reverse of step 1's reading, because at this instant it is between attempts rather than just after one. Both fields matter and neither is reliable alone.

   $kubectl get pod crasher -n tshNAME      READY   STATUS             RESTARTS        AGEcrasher   0/1     CrashLoopBackOff   6 (3m45s ago)   9m33s$kubectl describe pod crasher -n tsh | sed -n '/Events:/,$p' | tail -6  ----     ------     ----                   ----               -------  Normal   Scheduled  9m33s                  default-scheduler  Successfully assigned tsh/crasher to cka1001-node03  Normal   Pulled     3m45s (x7 over 9m32s)  kubelet            spec.containers{app}: Container image "busybox:1.36" already present on machine and can be accessed by the pod  Normal   Created    3m45s (x7 over 9m32s)  kubelet            spec.containers{app}: Container created  Normal   Started    3m45s (x7 over 9m32s)  kubelet            spec.containers{app}: Container started  Warning  BackOff    2s (x13 over 9m31s)    kubelet            spec.containers{app}: Back-off restarting failed container app in pod crasher_tsh(1bea9e6e-57db-43a0-bd5c-d0d015c329b7)$kubectl get pods -n tsh -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,REASON:.status.containerStatuses[0].state.waiting.reason,TERM:.status.containerStatuses[0].state.terminated.reason,RESTARTS:.status.containerStatuses[0].restartCount --no-headersbadimage   Pending   ImagePullBackOff   <none>      0crasher    Running   CrashLoopBackOff   <none>      6hungry     Failed    <none>             OOMKilled   0noauth     Pending   ImagePullBackOff   <none>      0nonode     Pending   <none>             <none>      <none>toobig     Pending   <none>             <none>      <none>

   Expected resultPulled/Created/Started all at x7, and a phase of Running for a Pod that has never worked. The other rows are the Pods from the neighbouring troubleshooting guides, kept because the comparison is the lesson.

   Success conditionYou can see phase: Running alongside a restart count of 6.

Troubleshooting

• A Pod shows Error and you were looking for CrashLoopBackOff.

  Why: The Pod cycles between Running, Error and CrashLoopBackOff. The status column is a snapshot.

  Fix:Use the restart count instead: kubectl get pods and read the RESTARTS column, or kubectl get pods --sort-by=.status.containerStatuses[0].restartCount to bring the worst to the bottom. A climbing count is the definition of a crash loop.

• kubectl logs --previous fails with unable to retrieve container logs.

  Why: The previous container's logs were collected by the runtime. Rapid restarts age them out quickly.

  Fix:Read the current attempt with plain kubectl logs; in a fast loop it fails the same way. If you need the very first failure, you have already lost it: for anything important, ship logs off the node. kubectl describe pod still shows the exit code and reason even when the logs are gone.

• Exit code 127 and the logs are empty.

  Why: Command not found. The container never got as far as running your program, so it produced no output of its own.

  Fix:Check command and args against what is actually in the image: kubectl run tmp --rm -it --image=<image> -- sh then look. A frequent cause is assuming a shell exists in a distroless or scratch image, so sh -c fails before anything else can. Another is command overriding the image's ENTRYPOINT and dropping a required wrapper.

• A container exits 0 and the Pod still restarts.

  Why: restartPolicy: Always, the default, restarts on any exit, success included.

  Fix:For something meant to run once, use a Job, or set restartPolicy: OnFailure or Never. Note the status will read Completed with a rising restart count, which looks contradictory and is exactly this. Init containers are the right tool for setup work that should run once and finish.

• Monitoring says everything is Running while an application is down.

  Why: .status.phase is Running for a Pod in a crash loop, as step 4 shows.

  Fix:Alert on the Ready condition or on restart counts, never on phase. kubectl get pods --field-selector=status.phase!=Running is a query that will miss every crash loop in the cluster; the READY 0/1 column is the honest one.

• The crash happens too fast to catch anything.

  Why: The container dies before you can exec into it, and each attempt produces little output.

  Fix:Two options. Temporarily replace command with ["sleep", "3600"] so the container stays up and you can kubectl exec in and run the real command by hand, which is the fastest way to see a configuration problem. Or use kubectl debug <pod> --copy-to=debug --set-image=... to work on a copy without touching the original.

Official sources

• Debug Pods
• Pod lifecycle: container states
• Restart policy
• Pod phase
• Debug running Pods