CertGrid CertGrid
Troubleshooting·Kubernetes and Cloud Native Associate

Reading Pod Status and Container State

Six broken Pods at once. One reports phase Running while restarting on a loop, one reports Failed with exit 137, and two sit Pending with no container status at all - because no container was ever created. Four statuses, and four different places the actual reason is written.

Troubleshooting and Debugging Guide 27 of 46 Beginner

Written against the versions above. Status strings and exit codes here are core Kubernetes and stable across versions. The scheduler's FailedScheduling wording changes between releases - the shape (how many nodes, and why each was rejected) does not.

Four nodes, each with 2 CPU allocatable - which is what makes the Insufficient cpu message in the last step a real number rather than an abstraction.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. Six broken Pods, and what the status column does not tell you

    One table, six failures. Read it carefully, because three things in it surprise almost everyone:

    badimage   Pending   ImagePullBackOff   <none>      0
crasher    Running   CrashLoopBackOff   <none>      6
hungry     Failed    <none>             OOMKilled   0
noauth     Pending   ImagePullBackOff   <none>      0
nonode     Pending   <none>             <none>      <none>
toobig     Pending   <none>             <none>      <none>

    crasher is phase Running. It has restarted six times and is failing every time. Phase is not health - a Pod is Running once a container has been started, and it stays Running while the kubelet keeps restarting a container that exits.

    hungry is phase Failed and its reason is under terminated, not waiting. A container that ran and stopped reports where it stopped; a container that never started reports what it is waiting for. Two different fields, and looking in the wrong one returns .

    nonode and toobig have no container status at all - not even a restart count. That is the important one. Both are Pending, like badimage, but for a completely different reason: the scheduler never placed them, so no container was ever created, so there is no container status to read. Their reason lives somewhere else entirely, which is the last step of this guide.

    So Pending covers two unrelated situations, and Running does not mean working. The status is the category. The cause is always one level down.

    bash Example session 
    kubectl get pods -n tsh -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,REASON:.status.containerStatuses[0].state.waiting.reason,TERM:.status.containerStatuses[0].state.terminated.reason,RESTARTS:.status.containerStatuses[0].restartCount --no-headersbadimage   Pending   ImagePullBackOff   <none>      0crasher    Running   CrashLoopBackOff   <none>      6hungry     Failed    <none>             OOMKilled   0noauth     Pending   ImagePullBackOff   <none>      0nonode     Pending   <none>             <none>      <none>toobig     Pending   <none>             <none>      <none>

    Expected resultSix Pods, four distinct statuses, and two rows with no container status.

    Success conditionYou can say why two Pending Pods have nothing in their container status.

  2. ImagePullBackOff: same status, different sentence

    ImagePullBackOff means the kubelet tried to pull an image, failed, and is waiting before trying again. It does not say why the pull failed - the message does.

    First Pod, a tag that does not exist:

    ... failed to resolve image: docker.io/library/nginx:this-tag-does-not-exist: not found

    Second Pod, same status, entirely different problem - a registry hostname that does not resolve:

    ... dial tcp: lookup registry.example.invalid on 127.0.0.53:53: no such host

    One is a typo in a tag, the other is DNS or a firewall. Same status, and the fix has nothing in common. Note also the second is still reporting ErrImagePull - the first attempt - where the first has already moved to ImagePullBackOff. They are the same failure at different points in the retry cycle, which is why treating them as different problems is a mistake.

    And kubectl logs cannot help here, which is worth seeing once:

    Error from server (BadRequest): container "badimage" in pod "badimage" is waiting to start: trying and failing to pull image

    There are no logs because there is no container. Reaching for logs first is the commonest wasted step in Kubernetes troubleshooting.

    bash Example session 
    sleep 25; kubectl get pod badimage -n tshNAME       READY   STATUS             RESTARTS   AGEbadimage   0/1     ImagePullBackOff   0          25skubectl get pod badimage -n tsh -o jsonpath="{.status.containerStatuses[0].state.waiting.reason}{\": \"}{.status.containerStatuses[0].state.waiting.message}{\"\n\"}"ImagePullBackOff: Back-off pulling image "nginx:this-tag-does-not-exist": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:this-tag-does-not-exist": failed to resolve image: docker.io/library/nginx:this-tag-does-not-exist: not foundsleep 30; kubectl get pod noauth -n tsh -o jsonpath="{.status.containerStatuses[0].state.waiting.reason}{\": \"}{.status.containerStatuses[0].state.waiting.message}{\"\n\"}"ErrImagePull: failed to pull and unpack image "registry.example.invalid/private/app:1.0": failed to resolve image: failed to do request: Head "https://registry.example.invalid/v2/private/app/manifests/1.0": dial tcp: lookup registry.example.invalid on 127.0.0.53:53: no such hostkubectl logs badimage -n tsh 2>&1 | tail -2Error from server (BadRequest): container "badimage" in pod "badimage" is waiting to start: trying and failing to pull image

    Expected resultTwo Pods with the same status class and two unrelated causes, and logs refusing to help.

    Success conditionYou read the message, not just the status.

  3. A container that exits: the logs are the app's own words

    This container prints a line, prints an error, and exits 3. Nine minutes later:

    crasher   0/1     CrashLoopBackOff   6 (3m45s ago)   9m33s

    CrashLoopBackOff is not a cause either - it is the kubelet saying "I have stopped retrying this quickly". Earlier in its life the same Pod showed Error; the status changes as the backoff grows. What actually happened is in the previous container's termination record:

    Error exit=3

    lastState, not state - the container currently in state is the one waiting to start again. And this is the one failure class where kubectl logs is exactly the right tool, because a container did run and it said why it gave up:

    starting up
config missing, giving up

    The events add nothing about the cause - Pulled, Created, Started, BackOff, over and over. That loop is the symptom. The application's own stderr is the diagnosis.

    bash Example session 
    kubectl get pod crasher -n tshNAME      READY   STATUS             RESTARTS        AGEcrasher   0/1     CrashLoopBackOff   6 (3m45s ago)   9m33skubectl get pod crasher -n tsh -o jsonpath="{.status.containerStatuses[0].lastState.terminated.reason}{\" exit=\"}{.status.containerStatuses[0].lastState.terminated.exitCode}{\"\n\"}"Error exit=3kubectl logs crasher -n tshstarting upconfig missing, giving upkubectl describe pod crasher -n tsh | sed -n '/Events:/,$p' | tail -6  Normal   Scheduled  9m33s                  default-scheduler  Successfully assigned tsh/crasher to cka1001-node03  Normal   Started    3m45s (x7 over 9m32s)  kubelet            spec.containers{app}: Container started  Warning  BackOff    2s (x13 over 9m31s)    kubelet            spec.containers{app}: Back-off restarting failed container app in pod crasher_tsh

    Expected resultCrashLoopBackOff with a non-zero exit code in lastState, and the application's reason in the logs.

    Success conditionYou found the cause in the container's own output, not in the events.

  4. OOMKilled is exit 137, and it is not a crash

    This container asks for 32Mi, is limited to 64Mi, and then writes 200Mi. It does not crash - it is killed:

    OOMKilled exit=137

    137 is 128 + 9: killed by signal 9. The kernel's OOM killer stopped the process because the container exceeded its memory *limit*. The application did nothing wrong and its logs will usually show nothing useful - it was terminated mid-work, with no chance to report anything.

    The fix is a number, and it is a specific one:

    limit=64Mi request=32Mi

    The limit is what gets you killed. The request is only what the scheduler used to place the Pod. Raise the limit, or make the workload use less - but know which of the two numbers you are changing, because raising the request changes where the Pod is scheduled and not whether it is killed.

    bash Example session 
    sleep 30; kubectl get pod hungry -n tshNAME     READY   STATUS      RESTARTS   AGEhungry   0/1     OOMKilled   0          30skubectl get pod hungry -n tsh -o jsonpath="{.status.containerStatuses[0].state.terminated.reason}{\" exit=\"}{.status.containerStatuses[0].state.terminated.exitCode}{\"\n\"}"OOMKilled exit=137kubectl get pod hungry -n tsh -o jsonpath="limit={.spec.containers[0].resources.limits.memory}{\" request=\"}{.spec.containers[0].resources.requests.memory}{\"\n\"}"limit=64Mi request=32Mi

    Expected resultReason OOMKilled, exit code 137, and the limit that caused it.

    Success conditionYou can name which of request and limit does the killing.

  5. Pending with no container: ask the scheduler instead

    Back to the two rows with no container status. Nothing is wrong with their images and no container has crashed, because the scheduler has not placed them on a node. There is nothing for the kubelet to report.

    The reason is on the Pod's own condition:

    PodScheduled=False Unschedulable: 0/4 nodes are available: 1 node(s) had
untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory.

    Read it as arithmetic. Four nodes; one is the control plane and carries a taint this Pod does not tolerate; the other three each lack the CPU and the memory. The Pod asked for 8 CPU and 32Gi, and every node has 2 CPU allocatable - so this is not a cluster problem, it is a request no node in this cluster could ever satisfy.

    The second Pending Pod produces the same status from a different sentence:

    0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 node(s)
didn't match Pod's node affinity/selector.

    No resource problem at all - a nodeSelector asking for a label no node carries. Same status, same empty container status, and the scheduler names both causes precisely if you ask it. For anything Pending, read the PodScheduled condition or the FailedScheduling event first; kubectl logs and kubectl describe's container section have nothing to offer yet.

    bash Example session 
    kubectl get pod toobig -n tsh -o jsonpath="{range .status.conditions[*]}{.type}={.status}{\" \"}{.reason}{\": \"}{.message}{\"\n\"}{end}"PodScheduled=False Unschedulable: 0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory. no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory --no-headerscka1001          2     3377972Kicka1001-node01   2     3377980Kicka1001-node02   2     3377980Kicka1001-node03   2     3377980Kikubectl describe pod nonode -n tsh | sed -n '/Events:/,$p' | tail -3  Type     Reason            Age   From               Message  ----     ------            ----  ----               -------  Warning  FailedScheduling  12s   default-scheduler  0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

    Expected resultTwo Unschedulable Pods, one short of resources and one short of a matching label, both named by the scheduler.

    Success conditionYou asked the scheduler rather than the kubelet.

Troubleshooting

Official sources

← All KCNA guides