Troubleshooting a Down Control Plane

Troubleshooting / Guide 94 of 103 / Advanced

• Kubernetes1.36.4
• Cluster1 node
• Runtimecontainerd 2.2.6
• CNICalico v3.32.1
• TimeAbout 30 min
• Reviewed21 August 2026

Written against the versions above. The manifest is moved aside and moved back. That is the supported way to stop a static Pod and the only reversible one.

Before you start

• The control plane guide, for static Pods and mirror Pods. This one depends on knowing why kubectl delete cannot stop the API server.
• The static Pods guide.
• Root on a disposable control plane node.

1. 1

   Stop the API server the only way that works

   Move the manifest out of /etc/kubernetes/manifests/ and the kubelet stops the container within seconds. That is the supported mechanism, and it is the only one that works: kubectl delete pod kube-apiserver-... deletes the mirror Pod and the kubelet republishes it immediately, as the control plane guide showed.

   Moving it to /root/parked/ rather than deleting it matters. The file is the only definition of your API server, including every flag: --etcd-servers, the certificate paths, --service-cluster-ip-range. Delete it and you are reconstructing that from documentation on a cluster you cannot query.

   Then kubectl is gone:

   The connection to the server 192.168.0.191:6443 was refused

   Every command. get, describe, logs, apply. There is no read-only fallback and no cached mode; kubectl is an API client and the API is not there.

   Note the exact wording, because it distinguishes this from its neighbours:

   • connection refused - nothing is listening on 6443. The API server process is not running.
   • Timeout / the server was unable to return a response - something is listening and cannot answer. The API server is up and its backend is not. That is the etcd guide.
   • Unauthorized - it answered and rejected you. Credentials, not availability.

   One word in that error tells you which of the three you are in.

   $sudo mkdir -p /root/parked && sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/parked/$kubectl get nodes 2>&1 | tail -1The connection to the server 192.168.0.191:6443 was refused - did you specify the right host or port?$kubectl cluster-info 2>&1 | tail -3 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.The connection to the server 192.168.0.191:6443 was refused - did you specify the right host or port?$sudo ls -1 /etc/kubernetes/manifests/etcd.yamlkube-controller-manager.yamlkube-scheduler.yaml

   Expected resultconnection refused on 6443 and three manifests where there were four. cluster-info suggesting cluster-info dump is unhelpful advice in this state: that command is also an API client.

   Success conditionkubectl fails with connection refused and the manifest directory has three files.

2. 2

   crictl is the view you have left

   crictl talks to containerd over the CRI socket. No API server, no kubelet required, no cluster. It is the tool for exactly this situation and it is worth being fluent in before you need it.

   Read crictl ps -a --name kube-apiserver:

   5bb054215798b  ...  2 minutes ago  Exited  kube-apiserver  2
   900f90392b45a  ...  6 minutes ago  Exited  kube-apiserver  1

   Two Exited containers, with ATTEMPT counts of 2 and 1. -a is essential: without it you see only running containers, and the one you care about is the one that stopped. The attempt counter is the equivalent of a Pod's restart count.

   crictl pods gives the sandbox view, and the useful column is STATE:

   etcd-cka4001                     Ready
   kube-apiserver-cka4001           NotReady
   kube-proxy-2chd6                 Ready
   kube-scheduler-cka4001           Ready
   kube-controller-manager-cka4001  Ready

   One NotReady among Readys, and it names itself. That single line is the fastest possible answer to "which control plane component is down" and it needs nothing but the node.

   crictl ps -a --state exited is the broader sweep: everything that has died recently, in one list. Note tigera-operator with 5 attempts in that output, which is a real consequence rather than noise. The operator cannot reach the API server either, so it is crashlooping too. Expect collateral: anything that talks to the API server starts failing, and those failures are downstream of the one that matters.

   The commands to have ready:

   • crictl ps -a - containers including dead ones.
   • crictl pods - sandboxes with their Ready state.
   • crictl logs - the container's output.
   • crictl inspect - the full container spec, including the exact command it was started with.

   $sudo crictl ps -a --name kube-apiserver 2>/dev/null | cut -c1-118CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT           5bb054215798b       b0f70fa6ec47e       2 minutes ago       Exited              kube-apiserver      2                 900f90392b45a       b0f70fa6ec47e       6 minutes ago       Exited              kube-apiserver      1                 $sudo crictl pods 2>/dev/null | cut -c1-104 | head -7POD ID              CREATED             STATE               NAME                                       N6377d6bab247f       3 minutes ago       Ready               etcd-cka4001                               k489afc17dafda       6 minutes ago       NotReady            kube-apiserver-cka4001                     k5892ed64bfc92       2 hours ago         Ready               kube-proxy-2chd6                           k492499ddf771b       3 hours ago         Ready               kube-scheduler-cka4001                     k71c04f780744c       3 hours ago         Ready               kube-controller-manager-cka4001            k8f4f7e8dfc343       15 hours ago        Ready               calico-apiserver-6b55b5fbd7-f89xq          c$sudo crictl ps -a --state exited 2>/dev/null | cut -c1-118 | head -4CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT     5bb054215798b       b0f70fa6ec47e       2 minutes ago       Exited              kube-apiserver            2           a3f55190fdca3       c49ec7e7876ff       6 minutes ago       Exited              tigera-operator           5           900f90392b45a       b0f70fa6ec47e       6 minutes ago       Exited              kube-apiserver            1           

   Expected resultThe API server's sandbox NotReady while every other component is Ready. Output is trimmed to 118 columns here; crictl tables are wider than a page.

   Success conditioncrictl pods shows exactly one NotReady sandbox and names it.

3. 3

   The kubelet's journal, read as a witness

   The kubelet is running and cannot reach the API server, so it logs every failure. Those lines are useful but they need reading carefully:

   "Failed to get status for pod" err="Get \"https://192.168.0.191:6443/...\":
   dial tcp 192.168.0.191:6443: connect: connection refused"

   That is the kubelet trying to post Pod status and being refused. It will repeat for every Pod on the node, several times a minute, and it will drown out everything else in the journal.

   The important point: these are symptoms, not the cause. The kubelet is healthy; it is reporting that its dependency is gone. Someone arriving at this journal cold could easily spend time on "the kubelet cannot connect" when the kubelet is the one component working correctly.

   So the order to read things in, on a node whose control plane is down:

   1. crictl pods - which component is NotReady. This is the answer, usually.
   2. crictl logs of that component - why it exited.
   3. journalctl -u kubelet - only if the component never started at all, because then the kubelet's own log holds the reason (a bad manifest, a missing mount, an unreadable certificate).

   That last case is the one worth separating. A YAML syntax error in kube-apiserver.yaml means the kubelet never creates a container, so crictl ps -a shows nothing for it and crictl logs has nothing to read. The evidence is in the kubelet's journal instead. No container at all points at the manifest; a container that exited points at its logs.

   $sudo journalctl -u kubelet -n 30 --no-pager 2>&1 | grep -iE 'apiserver|connection refused' | tail -3Aug 21 10:48:42 cka4001 kubelet[175688]: E0821 10:48:42.381965  175688 status_manager.go:1164] "Failed to get status for pod" err="Get \"https://192.168.0.191:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-cka4001\": dial tcp 192.168.0.191:6443: connect: connection refused" podUID="030f8892a3b3bf3983f52f056e586e1f" pod="kube-system/kube-controller-manager-cka4001"Aug 21 10:48:42 cka4001 kubelet[175688]: E0821 10:48:42.382069  175688 status_manager.go:1164] "Failed to get status for pod" err="Get \"https://192.168.0.191:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-cka4001\": dial tcp 192.168.0.191:6443: connect: connection refused" podUID="65dc2042cdf15359c233ba5c5a7f5a13" pod="kube-system/kube-scheduler-cka4001"Aug 21 10:48:42 cka4001 kubelet[175688]: E0821 10:48:42.382140  175688 status_manager.go:1164] "Failed to get status for pod" err="Get \"https://192.168.0.191:6443/api/v1/namespaces/calico-system/pods/calico-typha-5745db6759-l8xg5\": dial tcp 192.168.0.191:6443: connect: connection refused" podUID="030f8892a3b3bf3983f52f056e586e1f" pod="kube-system/kube-controller-manager-cka4001"

   Expected resultThe same refusal repeated for different Pods. Every line is the kubelet failing at something downstream of the real fault, which is why crictl pods is the better first command.

   Success conditionThe journal shows the kubelet being refused on 6443.

4. 4

   Put it back

   Move the manifest into the watched directory and the kubelet starts the container within seconds. /healthz answers ok, and kubectl works again. No restart of anything else, no repair.

   That recoverability is the point of static Pods and it is why moving the file is the right technique: the fix is exactly the inverse of the break.

   The recovery order for a real incident, which matters when more than one thing is wrong:

   1. etcd first. The API server cannot start without it, so an API server that will not come up on a cluster with a broken etcd is a symptom.
   2. Then the API server. Everything else needs it.
   3. Then the controller manager and scheduler. These can be down for a while without breaking running workloads: nothing new gets scheduled and nothing reconciles, but existing Pods keep serving. Less urgent than they feel.

   And what to check before you conclude the API server itself is at fault, since all three produce connection refused:

   • The manifest is present and valid. sudo ls /etc/kubernetes/manifests/ then read the file.
   • etcd is reachable. If not, the API server may be crashlooping for that reason.
   • The certificates are valid. sudo kubeadm certs check-expiration, though on a node with no API server that command reads the local files rather than the cluster.
   • Port 6443 is not taken by something else, which happens after a botched rebuild.

   One last note on backups. kubeadm keeps copies of the previous manifests under /etc/kubernetes/tmp/ after an upgrade, which is the file to reach for when an upgrade left a component unable to start and you need the version that worked.

   $sudo mv /root/parked/kube-apiserver.yaml /etc/kubernetes/manifests/$kubectl get --raw=/healthz; echook$kubectl get nodesNAME      STATUS   ROLES           AGE   VERSIONcka4001   Ready    control-plane   15h   v1.36.4$sudo ls -1 /root/parked/ 2>&1; echo '(empty means both manifests are back)'(empty means both manifests are back)

   Expected resultok from /healthz and a Ready node. The API server took under a minute to come back in this capture; poll /healthz rather than assuming, because it needs etcd up first.

   Success conditionkubectl get --raw=/healthz returns ok.

Troubleshooting

• kubectl fails with connection refused and you cannot see anything.

  Why: Nothing is listening on 6443. The API server process is not running.

  Fix:Go to the control plane node; kubectl cannot help. sudo crictl pods names the NotReady component, then sudo crictl logs <id> says why it exited. If no container exists for it at all, the kubelet never created one, so read journalctl -u kubelet for a manifest or mount error instead.

• The API server container is missing entirely from crictl ps -a.

  Why: The kubelet could not create it. A YAML error in the manifest, a missing hostPath, or an unreadable certificate.

  Fix:There are no container logs to read, so the kubelet's journal is the only source: journalctl -u kubelet -n 50. Validate the manifest before restoring it, and remember the file itself is the definition of every flag, so keep it rather than deleting it.

• The API server crashloops and its logs blame etcd.

  Why: The API server is stateless and cannot start without its datastore. This is a symptom of an etcd problem.

  Fix:Fix etcd first. sudo crictl pods | grep etcd for whether the sandbox is Ready, and sudo crictl logs on the etcd container. The etcd-unhealthy guide covers the case where etcd is up but not answering, which presents as a timeout rather than a refusal.

• You deleted the manifest instead of moving it.

  Why: The file was the only record of the API server's flags: etcd endpoints, certificate paths, service CIDR, advertise address.

  Fix:Recoverable, with effort. kubeadm init phase control-plane apiserver regenerates it from the cluster configuration, and after an upgrade a copy may exist under /etc/kubernetes/tmp/. The kubeadm-config ConfigMap in kube-system holds the cluster's settings, but reading it needs an API server, which is the problem. Move files, never delete them.

• The controller manager or scheduler is down and you are treating it as an outage.

  Why: Neither is in the request path. Running Pods keep serving; what stops is scheduling and reconciliation.

  Fix:Less urgent than it looks, and worth knowing so you fix things in the right order. Symptoms are Pods stuck Pending with no FailedScheduling event at all, and Deployments that never create ReplicaSets. Check the lease as well as the Pod: a component can be Running without holding its leader lease and therefore doing no work.

• Everything on the node looks healthy and kubectl still cannot connect.

  Why: Not the server. Your kubeconfig, the network path, or a firewall.

  Fix:Test from the node itself: sudo curl -k https://localhost:6443/healthz. Answering locally but not remotely makes it a network or --advertise-address problem, not a control plane one. kubectl config view --minify shows which endpoint you are actually pointed at, which on a machine with several contexts is worth confirming first.

Official sources

• Troubleshooting clusters
• Static Pods
• Debugging Kubernetes nodes with crictl
• kube-apiserver reference
• Kubernetes API health endpoints