CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Administrator

Troubleshooting Node NotReady

Stop a kubelet and watch what happens: the node goes NotReady with Ready=Unknown, two taints appear on their own, containers keep serving, and the Pod still reports Running because nothing is left to say otherwise.

Troubleshooting Guide 92 of 103 Intermediate

Written against the versions above. Done on a single-node cluster so stopping the kubelet breaks nothing else. On a worker the same sequence is safe.

Single-node cluster, deliberately. Everything runs on the node as root.
Server NameIP AddressOSRolesCPURAMHDD
CKA4001192.168.0.191Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB

Before you start

  1. Stop the kubelet, and notice what does not stop

    systemctl stop kubelet and the kubelet is gone. Now count the containers:

    18 still running. Including the workload Pod.

    That is the single most important fact about a NotReady node and it is the opposite of most people's intuition. The kubelet starts and supervises containers; it is not in the data path once they are running. Kill it and containerd carries on, the processes keep serving, the network keeps working.

    So a NotReady node is not necessarily a node that has stopped working. It is a node that has stopped reporting. Those are different problems and they need different urgency: an application may be perfectly healthy on a node the control plane has written off.

    What you lose immediately is control, not service. Nothing new can start there, nothing can be stopped cleanly, and no status can be updated.

    bash Example session 
    sudo systemctl stop kubelet && systemctl is-active kubeletinactivesudo crictl ps --no-trunc 2>/dev/null | wc -l18sudo crictl ps 2>/dev/null | grep -c survivor || echo 01

    Expected resultThe kubelet inactive and 18 containers still up, the workload among them. crictl is the runtime's own client and needs neither the kubelet nor the API server, which is why it still answers.

    Success conditioncrictl ps lists running containers while the kubelet is stopped.

  2. Unknown, not False, and two taints you did not add

    After the heartbeat timeout the node flips. Read the condition carefully, because the value is the diagnosis:

    Unknown NodeStatusUnknown: Kubelet stopped posting node status.

    Ready=Unknown, not Ready=False. The distinction is real and worth internalising:

    • Ready=False - the kubelet is running and reporting that the node is not usable. It has told you something. Look at what it said: disk, memory, a broken CNI.
    • Ready=Unknown - the kubelet is not reporting at all. Nobody knows anything. The node lifecycle controller in kube-controller-manager set this after the heartbeat stopped arriving, and the message says exactly that.

    Both display as NotReady in kubectl get nodes, which is why that column alone is not enough. Read the condition.

    Then the taints, which nobody added:

    node.kubernetes.io/unreachable=NoSchedule
node.kubernetes.io/unreachable=NoExecute

    The controller applies these automatically. NoSchedule stops new Pods being placed there. NoExecute starts evicting the Pods that are on it, after each Pod's tolerationSeconds (default 300, so five minutes).

    That five-minute delay is deliberate and worth knowing about, because it defines the window you have. A kubelet that comes back inside it costs nothing. Past it, the controller starts deleting Pod objects and rescheduling them elsewhere, and on a multi-node cluster that is when a brief node blip turns into a wave of rescheduling.

    bash Example session 
    kubectl get nodesNAME      STATUS     ROLES           AGE   VERSIONcka4001   NotReady   control-plane   15h   v1.36.4kubectl get node cka4001 -o jsonpath="{range .status.conditions[?(@.type=='Ready')]}{.status}{\" \"}{.reason}{\": \"}{.message}{\"\n\"}{end}"Unknown NodeStatusUnknown: Kubelet stopped posting node status.kubectl get node cka4001 -o jsonpath="{range .spec.taints[*]}{.key}={.effect}{\"\n\"}{end}"node.kubernetes.io/unreachable=NoSchedulenode.kubernetes.io/unreachable=NoExecute

    Expected resultUnknown with a message naming the cause, and two automatic taints. On a single-node cluster the eviction has nowhere to send anything, so the Pods stay; on a real cluster they would move.

    Success conditionThe Ready condition reads Unknown with reason NodeStatusUnknown.

  3. The Pod still says Running, and that is a lie

    This is the trap. Ask about the Pod:

    phase=Running ready=true

    The Pod reports healthy. It might be, or it might have crashed thirty seconds ago; there is no way to tell from here.

    The reason is that Pod status is written by the kubelet. With no kubelet, nothing updates it, so what you are reading is the last value posted before it stopped. It is a stale snapshot, and it will sit there unchanged for as long as the kubelet is down.

    So on a NotReady node, treat every Pod status as unverified. kubectl get pods telling you things are fine on such a node is not evidence of anything.

    The two commands after that show what you have actually lost:

    dial tcp 192.168.0.191:10250: connect: connection refused

    Both kubectl logs and kubectl exec fail, identically, on port 10250. That is the kubelet's own API, which the control plane guide introduced. The API server proxies those requests to the kubelet, and with no kubelet there is nothing listening.

    That gives you a precise diagnostic pair. On a NotReady node:

    • kubectl get still works, because it reads etcd through the API server and never touches the node.
    • kubectl logs, exec, port-forward, top all fail on 10250, because they need the kubelet.

    If get works and those fail, the kubelet is the problem. If everything fails, the API server is.

    Finally, systemctl status on the node states it plainly: Active: inactive (dead) since ... with a timestamp. That timestamp is worth reading, because it tells you whether the kubelet was stopped deliberately or died, and roughly when.

    bash Example session 
    kubectl get pod survivor -o jsonpath="phase={.status.phase}{\" ready=\"}{.status.containerStatuses[0].ready}{\"\n\"}"phase=Running ready=truekubectl logs survivor 2>&1 | tail -2Error from server: Get "https://192.168.0.191:10250/containerLogs/default/survivor/survivor": dial tcp 192.168.0.191:10250: connect: connection refusedkubectl exec survivor -- echo hi 2>&1 | tail -2error: unable to upgrade connection: error dialing backend: dial tcp 192.168.0.191:10250: connect: connection refusedsudo systemctl status kubelet --no-pager 2>&1 | head -5○ kubelet.service - kubelet: The Kubernetes Node Agent     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)    Drop-In: /usr/lib/systemd/system/kubelet.service.d             └─10-kubeadm.conf     Active: inactive (dead) since Fri 2026-08-21 10:46:26 UTC; 50s ago

    Expected resultA Pod claiming to be Running, and both node-proxied commands refused on 10250. Note the URL in the logs error: it names the port, the namespace, the Pod and the container, so it is unambiguous about what was attempted.

    Success conditionkubectl logs fails on port 10250 while kubectl get still works.

  4. Start it again

    systemctl start kubelet, wait, and the node is Ready. No repair, no rejoin, no kubeadm command: the kubelet reconnects, posts a heartbeat, and the controller clears the condition and removes the taints it added.

    That recovery being this cheap is worth knowing, because it shapes the right first move. When a node goes NotReady, try restarting the kubelet before anything else. It costs nothing if the containers are healthy, and it fixes a large share of real cases.

    What it does not fix is a kubelet that will not start, which is the next guide, or a node that is genuinely broken underneath. But it is the cheapest thing to try and it is often enough.

    One caution on the reverse direction. If the node was down long enough for NoExecute to evict its Pods, those Pods were deleted and recreated elsewhere. Bringing the kubelet back does not undo that, and the old containers on the returning node are cleaned up. So a node that has been away for more than five minutes comes back empty, and that is correct rather than a fault.

    bash Example session 
    sudo systemctl start kubelet && sleep 20 && systemctl is-active kubeletactivekubectl get nodesNAME      STATUS   ROLES           AGE   VERSIONcka4001   Ready    control-plane   15h   v1.36.4

    Expected resultReady again within seconds of the kubelet starting. The node's AGE is unchanged at 15h: the Node object was never deleted, only marked unreachable.

    Success conditionThe node returns to Ready after starting the kubelet.

Troubleshooting

Official sources

← All CKA guides