Troubleshooting Unhealthy etcd

Troubleshooting / Guide 95 of 103 / Advanced

• Kubernetes1.36.4
• etcd3.6.8
• Cluster1 node
• Runtimecontainerd 2.2.6
• TimeAbout 30 min
• Reviewed21 August 2026

Written against the versions above. etcd is stopped by moving its manifest, not by damaging its data. The data directory is untouched throughout.

Before you start

• The etcd backup and restore guide, which installs a version-matched etcdctl and covers the TLS flags.
• The control plane down guide, for the contrast this one is built on.
• Root on a disposable control plane node.

1. 1

   Healthy first, so the failure means something

   endpoint health on a working etcd, and note it does a real write:

   is healthy: successfully committed proposal: took = 3.374879ms

   "Committed proposal" is a Raft write that went through consensus and was applied. That makes it a genuine health check rather than a connection test: an etcd that accepts connections but cannot commit is unhealthy, and this command catches it.

   The timing is worth recording when things are working. 3.4ms is what normal looks like on this cluster. etcd's performance is dominated by disk fsync latency, and a cluster whose commits have crept from 3ms to 300ms is in trouble long before anything reports unhealthy. If you take one number from a healthy cluster, take this one.

   $sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint healthhttps://127.0.0.1:2379 is healthy: successfully committed proposal: took = 3.374879ms

   Expected resultHealthy, with a commit time in single-digit milliseconds. The four TLS flags are mandatory: etcd requires mutual TLS and will not answer without a client certificate.

   Success conditionendpoint health reports healthy with a commit time.

2. 2

   Stop etcd, and get a different failure

   Move etcd.yaml aside. Now endpoint health fails, and read how:

   is unhealthy: failed to commit proposal: context deadline exceeded
   Error: unhealthy cluster

   context deadline exceeded is a timeout, not a refusal. And then the important one:

   Error from server (Timeout): the server was unable to return a response in the time allotted,
   but may still be processing the request (get nodes)

   kubectl gets a Timeout, not connection refused. Compare the control plane down guide, where every command failed instantly with a refusal. The difference is diagnostic and it is the single most useful thing in this guide:

   • connection refused - nothing is listening on 6443. The API server is not running.
   • Timeout / unable to return a response - the API server is running, accepted your request, and could not answer. Its backend is the problem.

   The API server is stateless. It holds no data of its own, so with etcd gone it can still complete a TLS handshake, authenticate you, authorise you, and then fail at the last step when it tries to read. From the outside that looks like slowness rather than an outage, which is why this failure is often misread as load.

   A practical consequence: watch requests get slow before they fail. A degraded etcd raises latency across every API call, so the first symptom is usually a cluster that feels sluggish, kubectl taking seconds, and controllers falling behind. Refusals are obvious; timeouts hide.

   $sudo mv /etc/kubernetes/manifests/etcd.yaml /root/parked/$sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health 2>&1 | tail -2https://127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceededError: unhealthy cluster$kubectl get nodes 2>&1 | tail -2Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)$sudo crictl ps 2>/dev/null | grep -c kube-apiserver || echo 01

   Expected resultA timeout from kubectl, and the API server container still running. That count of 1 is the point: nothing crashed, so anything looking for a dead process finds none.

   Success conditionkubectl times out while the API server container is still running.

3. 3

   livez names the one thing that is wrong

   This is the command that turns a vague timeout into a precise answer. /livez?verbose breaks the API server's health into per-subsystem checks, and with etcd gone the output is unmistakable:

   [+]ping ok
   [+]log ok
   [+]loopback-serving-certificate ok
   [-]etcd failed: reason withheld
   [+]poststarthook/start-apiserver-admission-initializer ok
   ... every other check ok ...
   livez check failed

   Thirty-odd checks, one [-], and it names etcd. The API server is diagnosing itself and telling you exactly which dependency is down.

   That is why /livez?verbose belongs in your fingers for any control plane problem. It works whenever the API server is reachable at all, which is precisely the case where the symptom is a confusing timeout rather than an obvious refusal.

   Two details in that output worth explaining:

   reason withheld is deliberate, not a bug. The health endpoint is reachable without authentication, so it refuses to leak details of an internal failure to an anonymous caller. The specifics are in the API server's own logs.

   The response arrives as an error, Error from server (InternalError), with the whole check list embedded in the message. So a failing livez does not print cleanly; you read it out of an error string. Do not let that put you off, and note the trailing livez check failed which is the summary verdict.

   There are three endpoints and the distinction matters here: /livez is liveness (should this process be restarted), /readyz is readiness (should it receive traffic), and /healthz is the older combined check. For this failure /readyz also fails, and /livez is the one that tells you the process itself is not viable.

   $kubectl get --raw=/livez?verbose 2>&1 | tail -3Error from server (InternalError): an error on the server ("[+]ping ok\n[+]log ok\n[+]loopback-serving-certificate ok\n[-]etcd failed: reason withheld\n[+]poststarthook/start-apiserver-admission-initializer ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/priority-and-fairness-config-consumer ok\n[+]poststarthook/priority-and-fairness-filter ok\n[+]poststarthook/storage-object-count-tracker-hook ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/crd-informer-synced ok\n[+]poststarthook/start-system-namespaces-controller ok\n[+]poststarthook/peer-endpoint-reconciler-controller ok\n[+]poststarthook/start-cluster-authentication-info-controller ok\n[+]poststarthook/start-kube-apiserver-identity-lease-controller ok\n[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok\n[+]poststarthook/storage-readiness ok\n[+]poststarthook/start-legacy-token-tracking-controller ok\n[+]poststarthook/start-service-ip-repair-controllers ok\n[+]poststarthook/rbac/bootstrap-roles ok\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\n[+]poststarthook/priority-and-fairness-config-producer ok\n[+]poststarthook/bootstrap-controller ok\n[+]poststarthook/start-kubernetes-service-cidr-controller ok\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-status-local-available-controller ok\n[+]poststarthook/apiservice-status-remote-available-controller ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-discovery-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/apiservice-openapiv3-controller ok\nlivez check failed") has prevented the request from succeeding

   Expected resultOne [-]etcd failed among more than thirty [+] checks. The whole list arrives inside a single error string with literal \n separators, which is unpleasant to read and is exactly what the real output looks like.

   Success condition/livez?verbose shows [-]etcd failed and every other check ok.

4. 4

   The API server's own logs, and putting it back

   For the detail reason withheld held back, read the API server container's log. crictl logs works because it goes to the runtime, not the cluster:

   logger="etcd-client" msg="retrying of unary invoker failed"
   method="/etcdserverpb.KV/Range"
   error="... dial tcp 127.0.0.1:2379: connect: connection refused"

   The API server's embedded etcd client, retrying and failing. connection refused at the etcd layer, on port 2379, from inside the API server, while your kubectl request timed out on 6443. That is the two-layer picture in one place: refused underneath, timed out on top.

   The methods named are useful too. KV/Range is a read and KV/Txn is a write, so you can tell from the log whether reads, writes, or both are failing, which distinguishes a fully dead etcd from one in a read-only alarm state after hitting its space quota.

   Restoring is moving the manifest back. /livez?verbose then shows [+]etcd ok, the node is Ready, and the workload Pod is still Running because it never stopped: as the node NotReady guide showed, containers do not depend on the control plane to keep serving.

   The diagnostic summary worth carrying away, all three of which look like "the cluster is broken":

   • connection refused on 6443 - the API server is not running. Use crictl pods.
   • Timeout from the API server - the API server is running and etcd is not. Use /livez?verbose.
   • Slow but working - etcd is degraded rather than gone. Check commit latency with endpoint health and DB SIZE against its quota with endpoint status.

   The third is the one that gets missed, because nothing has failed yet.

   $sudo crictl logs $(sudo crictl ps -a --name kube-apiserver -q 2>/dev/null | head -1) 2>&1 | grep -iE 'etcd' | tail -3{"level":"warn","ts":"2026-08-21T10:50:01.764571Z","logger":"etcd-client","caller":"v3@v3.6.8/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14b57a07bc20/127.0.0.1:2379","method":"/etcdserverpb.KV/Range","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}{"level":"warn","ts":"2026-08-21T10:50:03.772807Z","logger":"etcd-client","caller":"v3@v3.6.8/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14b57b7e85a0/127.0.0.1:2379","method":"/etcdserverpb.KV/Txn","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}$sudo mv /root/parked/etcd.yaml /etc/kubernetes/manifests/$kubectl get --raw=/livez?verbose 2>&1 | head -5[+]ping ok[+]log ok[+]loopback-serving-certificate ok[+]etcd ok[+]poststarthook/start-apiserver-admission-initializer ok$kubectl get nodes; kubectl get pod survivor --no-headers | awk '{print $1, $3}'NAME      STATUS   ROLES           AGE   VERSIONcka4001   Ready    control-plane   15h   v1.36.4survivor Running

   Expected result[+]etcd ok where [-]etcd failed was, and the same livez output now printing cleanly rather than inside an error. The workload Pod is untouched.

   Success condition/livez?verbose shows [+]etcd ok and the node is Ready.

Troubleshooting

• kubectl times out rather than being refused, and you are looking for a crashed process.

  Why: The API server is running. Something it depends on is not, and etcd is the usual answer.

  Fix:kubectl get --raw='/livez?verbose' and look for the one [-] line. That works whenever the API server answers at all, which is exactly this case. connection refused means the opposite: use crictl pods on the node instead.

• livez says [-]etcd failed: reason withheld and you want the reason.

  Why: The health endpoint is unauthenticated, so it does not disclose internal failure detail.

  Fix:Read the API server's log: sudo crictl logs $(sudo crictl ps -a --name kube-apiserver -q | head -1) | grep etcd. Its embedded etcd client logs the underlying error, including whether reads (KV/Range) or writes (KV/Txn) are failing.

• etcd reports mvcc: database space exceeded and the cluster is read-only.

  Why: The database hit its quota, 2 GiB by default. etcd retains historical revisions, so the file grows well past live data.

  Fix:Three steps, all needed: compact to the current revision, defrag to return the space, then etcdctl alarm disarm to clear the alarm. Compaction alone does not shrink the file. Watch DB SIZE from endpoint status as routine rather than discovering this during an outage; the etcd backup guide shows a healthy cluster at 15 MB against a 2.1 GB quota.

• The cluster works but everything is slow.

  Why: etcd is degraded rather than down. Its latency is dominated by disk fsync, so a slow or contended disk raises the cost of every API call.

  Fix:The failure nobody notices until it becomes an outage. Compare endpoint health's commit time against a known-good figure; this cluster's healthy value is 3.4ms. Also check endpoint status for a raft term that keeps incrementing, which means repeated leader elections.

• A multi-member etcd cluster has lost quorum.

  Why: More than half the members are unavailable. Quorum is n/2 + 1, so a 3-member cluster tolerates one loss and no more.

  Fix:Without quorum etcd accepts no writes, so the whole cluster is read-only at best. etcdctl member list and endpoint status --cluster show which members are reachable. Recovery means restoring members until quorum returns, and the etcd backup guide covers why restoring several members from one snapshot separately does not work.

• etcd will not start after a restore or a move.

  Why: Usually member identity: the data directory's member name does not match the flags etcd is started with.

  Fix:sudo crictl logs on the etcd container names the mismatch. The etcd backup and restore guide covers the --name and --initial-cluster flags that have to agree with etcd.yaml. Note the data directory is authoritative once it exists, so --initial-* flags are ignored on subsequent starts.

Official sources

• Operating etcd clusters for Kubernetes
• Kubernetes API health endpoints
• etcd maintenance
• etcd disaster recovery
• Debugging Kubernetes nodes with crictl