Troubleshooting a Kubelet That Will Not Start

Troubleshooting / Guide 93 of 103 / Intermediate

• Kubernetes1.36.4
• Cluster1 node
• Runtimecontainerd 2.2.6
• CNICalico v3.32.1
• TimeAbout 25 min
• Reviewed21 August 2026

Written against the versions above. The config is backed up before being broken and restored afterwards. Do not practise this on a node you need.

Before you start

• The node NotReady guide, which covers what a stopped kubelet does to the cluster.
• Root on a disposable node.
• Basic systemd familiarity: systemctl status, journalctl -u.

1. 1

   Back it up, then break it

   Copy the config first. That is not ceremony: a kubelet with a broken config cannot start, and on a node you cannot otherwise reach, a backup is the difference between a one-line fix and a rebuild.

   The kubelet reads /var/lib/kubelet/config.yaml, which kubeadm writes. Appending an unclosed bracket gives a YAML syntax error, which stands in for the realistic causes: a hand-edited field, a bad --config flag, a botched upgrade, a typo in a drop-in.

   Note which file this is. There are three places kubelet configuration comes from and confusing them wastes time:

   • /var/lib/kubelet/config.yaml - the KubeletConfiguration itself, written by kubeadm from the cluster's kubelet-config ConfigMap. This is the one being broken here.
   • /var/lib/kubelet/kubeadm-flags.env - command-line flags kubeadm passes.
   • /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf - the systemd drop-in that ties them together.

   A syntax error in any of them stops the kubelet the same way, so check all three when the cause is not obvious.

   $sudo cp /var/lib/kubelet/config.yaml /root/kubelet-config.yaml.bak && echo savedsaved$sudo sh -c 'printf "thisIsNotAValidField: [unclosed\n" >> /var/lib/kubelet/config.yaml' && sudo tail -2 /var/lib/kubelet/config.yamlvolumeStatsAggPeriod: 0sthisIsNotAValidField: [unclosed

   Expected resultThe last real setting followed by the broken line. Keep the backup path in mind; step 4 uses it.

   Success conditionThe config file ends with the invalid line and a backup exists.

2. 2

   activating, which is the part that misleads

   Restart it, wait fifteen seconds, and ask systemd:

   activating

   Not failed. Not inactive. activating, which reads like a service that is still coming up and needs more time.

   It is not. systemctl status explains:

   Active: activating (auto-restart) (Result: exit-code) since ...; 4s ago

   auto-restart and Result: exit-code are the words that matter. The kubelet started, exited non-zero, and systemd is restarting it. The kubeadm drop-in sets Restart=always with a short RestartSec, so it loops indefinitely, and at any instant you look it is likely to be between attempts, which is when the state reads activating.

   So the trap is concrete: systemctl is-active kubelet returning activating means a crash loop, not a slow start. Waiting longer will not change it. The same shape as the CrashLoopBackOff guide's Pod, one layer down and reported by systemd instead of by Kubernetes.

   The reliable signals that this is a loop rather than a startup:

   • (auto-restart) in the Active line.
   • Result: exit-code rather than Result: success.
   • A since timestamp that keeps resetting each time you look.

   $sudo systemctl restart kubelet; sleep 15; systemctl is-active kubeletactivating$sudo systemctl status kubelet --no-pager 2>&1 | sed -n '2,7p'     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)    Drop-In: /usr/lib/systemd/system/kubelet.service.d             └─10-kubeadm.conf     Active: activating (auto-restart) (Result: exit-code) since Fri 2026-08-21 10:47:47 UTC; 4s ago Invocation: 466cbef88d2c4f91b491462e8b5d2eb2       Docs: https://kubernetes.io/docs/

   Expected resultactivating (auto-restart) (Result: exit-code). The Drop-In line naming 10-kubeadm.conf is worth noting: that file supplies the arguments, so it is one of the places a bad change can hide.

   Success conditionsystemctl status shows auto-restart and Result: exit-code.

3. 3

   journalctl names the file and the line

   systemctl status tells you it is failing. journalctl -u kubelet tells you why, and the kubelet is unusually good about this:

   "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml,
   error: ... failed to decode: yaml: line 50: did not find expected ',' or ']'"

   The path, the operation, and line 50. That is the whole diagnosis in one line, and it is why journalctl -u kubelet should be the first command you run on any node where the kubelet is not healthy, before systemctl status and long before anything in kubectl.

   The surrounding lines show the loop in systemd's own words: Failed with result 'exit-code', then Scheduled restart job, restart counter is at 1, then Started kubelet.service again. That counter incrementing is the clearest confirmation you are watching a loop.

   Flags worth having ready:

   • -u kubelet -n 50 - the last 50 lines. Start here.
   • -u kubelet -f - follow, to watch a restart happen.
   • -u kubelet --since '5 min ago' - bound it by time, which matters on a node that has been looping for hours and has thousands of near-identical lines.
   • -u kubelet -p err - errors only, which cuts a very noisy log down fast.

   One caution about reading these logs: a kubelet that cannot reach the API server also logs a great deal, and those messages are consequences rather than causes. Look for the first error after a restart, not the loudest or most frequent one.

   $sudo journalctl -u kubelet -n 8 --no-pager 2>&1 | tail -6Aug 21 10:47:37 cka4001 systemd[1]: kubelet.service: Failed with result 'exit-code'.Aug 21 10:47:47 cka4001 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.Aug 21 10:47:47 cka4001 systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.Aug 21 10:47:47 cka4001 kubelet[175657]: E0821 10:47:47.331467  175657 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to decode: yaml: line 50: did not find expected ',' or ']'"Aug 21 10:47:47 cka4001 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILUREAug 21 10:47:47 cka4001 systemd[1]: kubelet.service: Failed with result 'exit-code'.

   Expected resultA full loop iteration in six lines: failure, scheduled restart, start, the real error, exit, failure again. The restart counter is at 1 will keep climbing.

   Success conditionjournalctl names the config file and the line number.

4. 4

   Restore and confirm

   Copy the backup back, restart, and the kubelet is active. The node returns to Ready on its own once a heartbeat lands.

   The sequence in full, as a habit to carry into a real incident:

   1. journalctl -u kubelet -n 50 - get the actual error. Do this first.
   2. Fix what it names. Usually a config file, a certificate, or a missing /var/lib/kubelet path.
   3. systemctl restart kubelet and re-read the journal, rather than assuming.
   4. kubectl get nodes to confirm the node came back.

   And the causes worth having in mind before you start reading, roughly by how often they turn up:

   • Config syntax or an unknown field, as here. Named precisely in the log.
   • An expired kubelet client certificate. The kubelet starts but cannot authenticate, so the node stays NotReady while the service reports active. That has its own guide.
   • The container runtime is down. The kubelet cannot reach the CRI socket and logs connection errors against /run/containerd/containerd.sock. Check systemctl status containerd too.
   • swap enabled where the configuration does not permit it, which is a hard refusal at startup.
   • A version skew after a partial upgrade, where the config on disk uses a field the installed kubelet does not know.

   The first of those is a syntax problem and the rest are environment problems, and journalctl distinguishes them immediately. That is the reason to read it before touching anything.

   $sudo cp /root/kubelet-config.yaml.bak /var/lib/kubelet/config.yaml && sudo systemctl restart kubelet && sleep 20 && systemctl is-active kubeletactive$kubectl get nodesNAME      STATUS   ROLES           AGE   VERSIONcka4001   Ready    control-plane   15h   v1.36.4

   Expected resultactive, and the node Ready. Compare against step 2: same command, and the difference between active and activating is the whole story.

   Success conditionsystemctl is-active kubelet returns active and the node is Ready.

Troubleshooting

• systemctl is-active kubelet says activating and you are waiting for it to finish.

  Why: It is in a restart loop, not starting up. The kubeadm drop-in sets Restart=always, so a failing kubelet cycles indefinitely.

  Fix:Do not wait. Confirm with systemctl status kubelet and look for (auto-restart) and Result: exit-code, then read journalctl -u kubelet -n 50 for the real error. A genuinely starting kubelet reaches active in a few seconds.

• The journal is thousands of near-identical lines and you cannot find the cause.

  Why: A looping kubelet logs the same failure every few seconds, and a kubelet that cannot reach the API server logs a great deal that is consequence rather than cause.

  Fix:Bound it and filter: journalctl -u kubelet --since '5 min ago' -p err. Then find the first error after a Started kubelet.service line; everything after it in that iteration is downstream. grep -m1 after a restart marker is the quick version.

• The kubelet is active but the node stays NotReady.

  Why: It is running and cannot register. Usually certificates or connectivity to the API server, not the kubelet itself.

  Fix:A different failure from this guide's: the process is up. Read the journal for authentication or connection errors, check sudo kubeadm certs check-expiration, and confirm the node can reach the API server's advertised address on 6443. The expired-certificates guide covers the certificate case.

• The kubelet fails with an error about the container runtime.

  Why: containerd is down or its socket has moved, so the kubelet cannot talk to the CRI.

  Fix:Check the layer below before the kubelet: systemctl status containerd and sudo crictl version. The reset-and-rejoin guide notes that kubeadm reset leaves the socket briefly unavailable, which produces exactly this transiently. Restart containerd, wait for crictl version to answer, then restart the kubelet.

• It broke immediately after an upgrade.

  Why: A config on disk using a field the newly installed kubelet does not recognise, or the reverse after a partial rollback.

  Fix:The journal names the field. kubeadm upgrade node rewrites /var/lib/kubelet/config.yaml from the cluster's kubelet-config ConfigMap, and skipping that step is the usual reason for a mismatch; the upgrade guide covers it. kubeadm keeps a backup of the previous config under /etc/kubernetes/tmp/.

• You cannot reach the node to read the journal.

  Why: No SSH, and kubectl cannot help because the node is not reporting.

  Fix:kubectl debug node/<name> --image=busybox -it -- sh creates a Pod on that node with the host filesystem at /host, which works whenever the kubelet is running enough to start a Pod. If it is not, you need console access; there is no in-cluster route to a node whose kubelet is dead.

Official sources

• Troubleshooting kubeadm
• kubelet configuration file
• KubeletConfiguration reference
• Debug cluster nodes
• kubelet reference