CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Administrator

Cluster Triage Order

Under pressure, the order you check things in matters more than knowing every command. Work outward from the layer everything else depends on: nodes, then control plane, then warnings, then capacity, then the workload. Four commands cover the first four layers.

Troubleshooting Guide 75 of 103 Beginner

Written against the versions above. The order is the transferable part. `/livez` and `/readyz` are only reachable on a cluster whose API server is up, which is why the order starts below them.

A healthy four-node cluster with two deliberately broken Pods, so the sweep has something real to find.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. Why order beats knowledge

    The failures that take longest to fix are rarely the hardest. They are the ones where someone spent forty minutes reading application logs while a node was NotReady, or debugged a Service while the CNI was down on one node.

    The fix is to work from the bottom up, because each layer depends on the one below it:

    1. Nodes. Nothing runs on a node that is not Ready.
    2. Control plane. If the API server or the scheduler is unhealthy, the cluster is not reconciling and every symptom above is downstream of that.
    3. Warnings. The cluster has usually already written down what is wrong.
    4. Capacity. A cluster that is full behaves like a broken one, and the symptoms look nothing like "full".
    5. The workload itself. Only now.

    The discipline is not to skip ahead. A report of "the app is down" is an invitation to look at the app, and looking at nodes first costs ten seconds and rules out an entire class of cause.

    One caveat that shapes everything below: all of these commands need a working API server. If kubectl itself is timing out, you are in a different situation, and the Control plane down guide covers it from the node, where crictl and the kubelet logs replace kubectl.

    bash Example session 
    kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type,VERSION:.status.nodeInfo.kubeletVersion --no-headerscka1001          Ready   v1.36.4cka1001-node01   Ready   v1.36.4cka1001-node02   Ready   v1.36.4cka1001-node03   Ready   v1.36.4

    Expected resultEvery node Ready on the same version. The version column matters more than it looks: a node left behind by a partial upgrade is a common source of behaviour that differs by node.

    Success conditionYou can list nodes with status and kubelet version in one command.

  2. Step 1, nodes: read the conditions, not just the word Ready

    Ready in kubectl get nodes is a summary. The conditions behind it are where the useful detail is:

    Type                 Status  Reason                       Message
NetworkUnavailable   False   CalicoIsUp                   Calico is running on this node
MemoryPressure       False   KubeletHasSufficientMemory   kubelet has sufficient memory available
DiskPressure         False   KubeletHasNoDiskPressure     kubelet has no disk pressure
PIDPressure          False   KubeletHasSufficientPID      kubelet has sufficient PID available

    Read this carefully, because the polarity is deliberately confusing. For the four pressure and network conditions, False is healthy. They are named for the problem, so DiskPressure=False means there is no disk pressure. Ready is the one that inverts: Ready=True is healthy.

    Each of those conditions produces a distinct failure that does not look like a node problem from above:

    • DiskPressure=True and the kubelet starts evicting Pods and refusing new ones. The symptom is Pods disappearing or going Pending with no obvious cause, and the cause is a full disk on one node, often from image or log accumulation.
    • MemoryPressure=True and the same, with eviction ordered by QoS class, so BestEffort Pods vanish first.
    • NetworkUnavailable=True means the CNI has not initialised. Pods schedule and never get an IP. This is the one that most often gets mistaken for a Service or DNS problem.
    • PIDPressure=True is rare, and when it happens something is forking without limit.

    LastHeartbeatTime is the other field worth a look. The kubelet updates it every few seconds; a heartbeat minutes old on a node still showing Ready means the node stopped reporting and the controller manager has not yet flipped it, which it does after node-monitor-grace-period. So a node can be genuinely gone and still read Ready for a short window, and the heartbeat is what gives it away.

    The practical form of this step: kubectl get nodes first, and kubectl describe node on anything that is not plainly Ready.

    bash Example session 
    kubectl describe node cka1001-node01 | grep -A6 '^Conditions:' | head -8Conditions:  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message  ----                 ------  -----------------                 ------------------                ------                       -------  NetworkUnavailable   False   Fri, 21 Aug 2026 04:50:40 +0000   Fri, 21 Aug 2026 04:50:40 +0000   CalicoIsUp                   Calico is running on this node  MemoryPressure       False   Fri, 21 Aug 2026 11:59:33 +0000   Fri, 21 Aug 2026 04:49:26 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available  DiskPressure         False   Fri, 21 Aug 2026 11:59:33 +0000   Fri, 21 Aug 2026 04:49:26 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure  PIDPressure          False   Fri, 21 Aug 2026 11:59:33 +0000   Fri, 21 Aug 2026 04:49:26 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available

    Expected resultFour conditions, all False, which is the healthy state for all four. Note the heartbeat times are seconds old while the transition times are from when the node joined.

    Success conditionYou can read a node's conditions and know which polarity means healthy.

  3. Step 2, control plane: /livez is the fastest answer

    The API server exposes its own health, broken down by subsystem:

    [+]poststarthook/apiservice-discovery-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
livez check passed

    Three endpoints, and the distinction between them is worth knowing:

    • /livez - is the API server alive? Failing means it should be restarted.
    • /readyz - is it ready to serve? Failing means take it out of the load balancer, but do not restart it.
    • /healthz - the older combined endpoint, kept for compatibility. Prefer the other two.

    ?verbose is what makes them useful. Without it you get ok; with it, one line per check, and a failing check is marked [-] with its name. An etcd problem shows up as [-]etcd failed, which points straight at the layer below rather than at the API server itself. Full output is long, so | grep -v ' ok$' is the practical filter when you are looking for a failure.

    Then the static Pods, which are the rest of the control plane:

    etcd-cka1001                      true   13
kube-apiserver-cka1001            true   0
kube-controller-manager-cka1001   true   0
kube-scheduler-cka1001            true   0

    Two columns to read. Ready should be true for all of them. Restarts should be stable, and the number itself matters less than whether it is climbing: a control plane component with a growing restart count is crash-looping, and that is a much worse state than a high count that has not moved in days. The 13 on etcd in this output is from earlier deliberate breakage in the etcd guide, and it has been flat since.

    Which components you can even see depends on the layer: kube-proxy, CoreDNS and the CNI are also here and are equally capable of breaking everything above them. A CoreDNS Pod not Ready explains every "cannot resolve" report in the cluster at once.

    kubectl cluster-info confirms which endpoint you are actually talking to:

    Kubernetes control plane is running at https://192.168.0.175:6443

    A trivial check that occasionally saves a lot of time, because a kubeconfig pointing at the wrong cluster produces symptoms that make no sense in the cluster you think you are looking at.

    Note that kubectl get componentstatuses is deprecated and reports nothing useful on modern clusters. If you learned it from an older guide, replace it with /livez?verbose and the static Pod listing.

    bash Example session 
    kubectl get --raw='/livez?verbose' 2>&1 | tail -6[+]poststarthook/apiservice-discovery-controller ok[+]poststarthook/kube-apiserver-autoregistration ok[+]autoregister-completion ok[+]poststarthook/apiservice-openapi-controller ok[+]poststarthook/apiservice-openapiv3-controller oklivez check passedkubectl get --raw='/readyz' ; echookkubectl get pods -n kube-system -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready,RESTARTS:.status.containerStatuses[0].restartCount --no-headers | head -12coredns-589f44dc88-fdcml          true   0coredns-589f44dc88-fkjxx          true   0etcd-cka1001                      true   13kube-apiserver-cka1001            true   0kube-controller-manager-cka1001   true   0kube-proxy-bxs9g                  true   0kube-proxy-mzbdk                  true   0kube-proxy-tmb4l                  true   0kube-proxy-wpjgh                  true   0kube-scheduler-cka1001            true   0metrics-server-5b58578978-l7s8x   true   0kubectl cluster-info | head -4Kubernetes control plane is running at https://192.168.0.175:6443CoreDNS is running at https://192.168.0.175:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

    Expected resultlivez check passed, readyz returning ok, and every kube-system Pod Ready. One kube-proxy per node is the expected count, four here.

    Success conditionYou can check API server health by subsystem and confirm the control plane Pods are Ready.

  4. Step 3, warnings: the cluster has already written it down

    Two commands, and they find real problems without knowing anything about what is deployed:

    ingress-nginx   FailedMount        ingress-nginx-controller-7df97f6c86-bz2n5
tri             Failed             bad-77dc8dbb75-krx5k
tri             FailedScheduling   toobig

    The cluster-wide warning sweep. Three problems in three namespaces, found without being told where to look. -A matters: the ingress-nginx one is in a namespace nobody was thinking about.

    Then the Pod sweep, which uses a field selector rather than events, so it is not subject to the one-hour event TTL:

    tri   bad-77dc8dbb75-krx5k   Pending
tri   toobig                 Pending

    --field-selector=status.phase!=Running,status.phase!=Succeeded is the useful form. Running is working, Succeeded is a completed Job, and everything else deserves a look. It scales: on a cluster with 3,000 Pods this returns the handful that need attention.

    One subtlety in that output. Both Pods are Pending here, but for entirely different reasons: bad was scheduled and cannot pull its image, toobig was never scheduled at all. Phase does not distinguish them; the events in step 3's first command do, Failed against FailedScheduling. Which is why the two commands go together, and why the events guide is worth reading alongside this one.

    Sort the events by time when there are many, --sort-by=.lastTimestamp, and remember the TTL: an incident older than an hour has no events left, so absence of warnings is not evidence of health.

    bash Example session 
    kubectl get events -A --field-selector type=Warning -o custom-columns=NS:.metadata.namespace,REASON:.reason,OBJECT:.involvedObject.name --no-headers 2>/dev/null | sort -u | head -8ingress-nginx   FailedMount        ingress-nginx-controller-7df97f6c86-bz2n5tri             Failed             bad-77dc8dbb75-krx5ktri             FailedScheduling   toobigkubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase --no-headerstri   bad-77dc8dbb75-krx5k   Pendingtri   toobig                 Pending

    Expected resultWarnings from three namespaces including one nobody deployed to deliberately, and the two Pods that are not Running. Both broken Pods show the same phase for different reasons.

    Success conditionA single command finds every warning in the cluster, and another finds every Pod that is not Running.

  5. Step 4, capacity: full clusters do not say they are full

    Two views, and they answer different questions.

    Actual usage, from metrics-server:

    cka1001          124m   6%    1414Mi   42%
cka1001-node01   30m    1%    863Mi    26%
cka1001-node02   29m    1%    865Mi    26%
cka1001-node03   29m    1%    689Mi    20%

    Requests, from the scheduler's point of view:

    Resource           Requests   Limits
cpu                100m (5%)  0 (0%)
memory             70Mi (2%)  170Mi (5%)

    These two numbers are frequently far apart, and the gap is the single most misunderstood thing about Kubernetes capacity. The scheduler places Pods using requests, not usage. A node at 5% CPU usage whose requests total 100% will not accept another Pod, and kubectl top will show it nearly idle. Somebody looking at top concludes there is plenty of room, while the scheduler correctly refuses to place anything.

    So when Pods are Pending with Insufficient cpu, look at requests. kubectl describe node is the direct way; the Allocated resources block is exactly that accounting.

    The reverse gap matters too. Requests well below usage means nodes are overcommitted, and the failure mode is OOM kills and CPU throttling under load rather than anything visible at rest. Both gaps are worth knowing about, and neither is visible from a single view.

    When kubectl top returns error: Metrics API not available, metrics-server is not installed or not working. That is worth knowing on its own, because a broken metrics-server also breaks every HorizontalPodAutoscaler in the cluster, silently.

    bash Example session 
    kubectl top nodes --no-headers 2>&1 | head -5cka1001          124m   6%    1414Mi   42%   cka1001-node01   30m    1%    863Mi    26%   cka1001-node02   29m    1%    865Mi    26%   cka1001-node03   29m    1%    689Mi    20%   kubectl describe node cka1001-node01 | sed -n '/Allocated resources/,/Events/p' | head -8Allocated resources:  (Total limits may be over 100 percent, i.e., overcommitted.)  Resource           Requests   Limits  --------           --------   ------  cpu                100m (5%)  0 (0%)  memory             70Mi (2%)  170Mi (5%)  ephemeral-storage  0 (0%)     0 (0%)  hugepages-1Gi      0 (0%)     0 (0%)

    Expected resultUsage from metrics-server and requests from the node's own accounting. Both low on this cluster, which is why the toobig Pod's 40 CPUs failed on capacity rather than on anything being busy.

    Success conditionYou can read usage and requests separately and know which one the scheduler uses.

  6. The whole sweep in one command

    Compressed to what fits on a screen, which is what to run before anything else:

    cka1001 Ready
cka1001-node01 Ready
cka1001-node02 Ready
cka1001-node03 Ready
tri bad-77dc8dbb75-krx5k ImagePullBackOff
tri toobig Pending

    Six lines: every node accounted for, and every Pod in the cluster that is not Running or Completed. Two seconds, and the shape of the problem is already visible.

    Note this view shows ImagePullBackOff where the phase-based command in step 3 showed Pending. kubectl get pods prints the container's waiting reason in the STATUS column rather than the raw phase, which is more useful, and it is also why the two commands appear to disagree. They do not; one is reading .status.phase and the other is reading the container state.

    What the sweep tells you next:

    • A node not Ready stops here. Fix that first; the Pod list is downstream of it. Node NotReady and Kubelet will not start are the guides.
    • Everything Ready and Pods failing means the platform is fine and this is a workload problem. Read the reason in the STATUS column and go to the guide for it: ImagePullBackOff, CrashLoopBackOff, Pod stuck Pending, OOMKilled.
    • Nothing wrong in the sweep and users still reporting problems means the failure is inside a Running, Ready Pod, or in the path to it. Then it is DNS, Services, NetworkPolicy or Ingress, in that order, and the network troubleshooting guides pick it up.

    The habit worth building is running this sweep before forming a theory. It is the difference between an hour of debugging an application and ten seconds of noticing a node went away.

    bash Example session 
    kubectl get nodes --no-headers | awk '{print $1, $2}'; kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed" {print $1, $2, $4}'cka1001 Readycka1001-node01 Readycka1001-node02 Readycka1001-node03 Readytri bad-77dc8dbb75-krx5k ImagePullBackOfftri toobig Pending

    Expected resultNodes and problem Pods in one output. The $4 in the awk is the STATUS column of kubectl get pods -A, so the filter drops healthy and completed Pods.

    Success conditionOne command gives you node health and every Pod that needs attention.

Troubleshooting

Official sources

← All CKA guides