CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Administrator

Troubleshooting Pod Pending

The scheduler explains itself precisely and almost nobody reads it. Two unschedulable Pods, two different tallies, and the reason every container-status query returns nothing for either of them.

Troubleshooting Guide 80 of 103 Beginner

Written against the versions above. Node counts in the message are this lab's four. The format of the tally is what to learn.

Four nodes: one control plane with a taint, three workers with 2 cores each.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. Not enough room, and the tally that says so

    A Pod requesting 8 CPUs and 32Gi on a cluster of 2-core nodes. It stays Pending, and the scheduler explains exactly why:

    0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory.

    That sentence is a tally, and reading it properly is the whole skill. It says: four nodes considered, none usable; one rejected on a taint, three rejected on CPU and the same three on memory.

    Two things to take from the format:

    • The counts should add up to the total, allowing for a node being rejected on several grounds. Here 1 tainted plus 3 short of resources covers all four.
    • The reasons are ranked by frequency, not by importance, so the first one listed is not necessarily your problem. 1 node(s) had untolerated taint(s) is the control plane's NoSchedule taint doing its job; it appears in almost every unschedulable message on a kubeadm cluster and is usually noise.

    The real cause here is 3 Insufficient cpu, confirmed by the allocatable figures: every node has 2 cores, and the Pod wants 8. No amount of waiting fixes that.

    The trailing clause matters too: preemption: 0/4 nodes are available: 4 Preemption is not helpful. The scheduler considered evicting lower-priority Pods to make room and concluded it would not help, which is correct when the request exceeds the node's total capacity. Seeing Preemption is not helpful tells you the request is beyond any single node, not merely beyond what is free right now.

    A note on where to read this. The Unschedulable condition holds the same text as the event, and it persists, while events expire after an hour. On a Pod that has been Pending since yesterday the event list is empty and the condition still tells you why.

    bash Example session 
    kubectl apply -f - <<'EOF'apiVersion: v1kind: Podmetadata:  name: toobig  namespace: tshspec:  containers:  - name: app    image: nginx:1.29-alpine    resources:      requests: {cpu: "8", memory: 32Gi}EOFpod/toobig createdsleep 15; kubectl get pod toobig -n tshNAME     READY   STATUS    RESTARTS   AGEtoobig   0/1     Pending   0          15skubectl get pod toobig -n tsh -o jsonpath="{range .status.conditions[*]}{.type}={.status}{\" \"}{.reason}{\": \"}{.message}{\"\n\"}{end}"PodScheduled=False Unschedulable: 0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory. no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.kubectl describe pod toobig -n tsh | sed -n '/Events:/,$p' | tail -4Events:  Type     Reason            Age   From               Message  ----     ------            ----  ----               -------  Warning  FailedScheduling  15s   default-scheduler  0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory. no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory --no-headerscka1001          2     3377972Kicka1001-node01   2     3377980Kicka1001-node02   2     3377980Kicka1001-node03   2     3377980Ki

    Expected resultThe same message from both the condition and the event. allocatable rather than capacity is the right column to compare against: it is capacity minus what the kubelet reserves for the system, so it is what the scheduler actually has to spend.

    Success conditionThe Unschedulable message names Insufficient cpu.

  2. A label nothing has, and a different tally

    Same Pending status, tiny resource request, completely different cause. This Pod asks for disktype: nvme-that-does-not-exist.

    0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 node(s) didn't match Pod's node affinity/selector.

    The taint clause is identical noise. The informative half changed to didn't match Pod's node affinity/selector, which covers nodeSelector, nodeAffinity and nodeName together, so the message does not tell you which of the three. Read the spec to find out.

    So a short field guide to the common clauses:

    • Insufficient cpu / Insufficient memory - requests exceed what is free. Compare against allocatable.
    • didn't match Pod's node affinity/selector - a label requirement nothing satisfies. Check with kubectl get nodes -l ; no output means no candidate.
    • had untolerated taint(s) - usually the control plane, and usually irrelevant. It becomes the real cause when it accounts for all nodes.
    • node(s) had volume node affinity conflict - the Pod's PersistentVolume is node-local and that node is not usable.
    • didn't match pod anti-affinity rules - the Pod refuses to sit with something already there.
    • node(s) were unschedulable - nodes are cordoned. kubectl get nodes shows SchedulingDisabled.

    The method in all cases is the same: subtract the noise, then check whether the remaining reason accounts for every node. If it does, that is your answer.

    bash Example session 
    kubectl apply -f - <<'EOF'apiVersion: v1kind: Podmetadata:  name: nonode  namespace: tshspec:  nodeSelector:    disktype: nvme-that-does-not-exist  containers:  - name: app    image: nginx:1.29-alpine    resources: {requests: {cpu: 10m, memory: 16Mi}}EOFpod/nonode createdsleep 12; kubectl get pod nonode -n tsh --no-headers | awk '{print $1, $3}'nonode Pendingkubectl describe pod nonode -n tsh | sed -n '/Events:/,$p' | tail -3  Type     Reason            Age   From               Message  ----     ------            ----  ----               -------  Warning  FailedScheduling  12s   default-scheduler  0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

    Expected resultA 10m CPU request rejected, which rules resources out immediately and shows that Pending says nothing about size on its own.

    Success conditionThe message names an affinity or selector mismatch.

  3. Why every container query comes back empty

    Now the detail that wastes the most time, and it is visible in the comparison table.

    nonode   Pending   <none>   <none>   <none>
toobig   Pending   <none>   <none>   <none>

    Every column reads , including the restart count. Compare badimage, also Pending, which has a real waiting reason.

    The difference is that an unscheduled Pod has no containerStatuses at all. That array is populated by the kubelet, and no kubelet owns this Pod yet, because it has no node. So there is nothing to report and nothing to read.

    That divides Pending into two genuinely different situations:

    • Pending, no container status - the scheduler could not place it. Look at .status.conditions and FailedScheduling events.
    • Pending, with a container status - it was placed and the kubelet cannot start it. Image pull, volume mount, or a config reference that does not resolve.

    kubectl get pod -o jsonpath='{.spec.nodeName}' settles it in one command: empty means unscheduled.

    The practical consequence for tooling: any script that inspects .status.containerStatuses[0] silently returns nothing for every unschedulable Pod in the cluster. A dashboard built that way shows those Pods as having no problem at all.

    One last thing the event list shows, and it is a small relief: there is exactly one FailedScheduling event per Pod rather than a stream. The scheduler retries with back-off and updates the existing event's count rather than emitting new ones, so a Pending Pod does not flood the event log. It also means the Age on that event is when it was last retried, not when the trouble started.

    bash Example session 
    kubectl get pods -n tsh -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,REASON:.status.containerStatuses[0].state.waiting.reason,TERM:.status.containerStatuses[0].state.terminated.reason,RESTARTS:.status.containerStatuses[0].restartCount --no-headersbadimage   Pending   ImagePullBackOff   <none>      0crasher    Running   <none>             Error       4hungry     Failed    <none>             OOMKilled   0noauth     Pending   ImagePullBackOff   <none>      0nonode     Pending   <none>             <none>      <none>toobig     Pending   <none>             <none>      <none>kubectl get events -n tsh --sort-by=.lastTimestamp -o custom-columns=OBJ:.involvedObject.name,TYPE:.type,REASON:.reason --no-headers | tail -12noauth     Warning   Failednoauth     Normal    Pullingtoobig     Warning   FailedSchedulingnonode     Warning   FailedSchedulingbadimage   Warning   Failedbadimage   Normal    BackOffnoauth     Normal    BackOffnoauth     Warning   Failedcrasher    Warning   BackOffcrasher    Normal    Startedcrasher    Normal    Createdcrasher    Normal    Pulled

    Expected resultTwo Pending rows with nothing in them and two with a reason. The absence of Scheduled events for toobig and nonode, where every other Pod has one, is the same fact from the event side.

    Success conditionYou can tell an unscheduled Pod from an unstarted one by whether container status exists.

Troubleshooting

Official sources

← All CKA guides