Kubernetes troubleshooting cheat sheet

Nothing runs on a node that is not Ready. Ten seconds, and it rules out a whole class of cause.

Also read the version column: a node left behind by a partial upgrade explains behaviour that differs by node.

Every Pod in the cluster that is neither working nor finished. Scales to thousands.

Reads current state, so unlike events it is not subject to the one-hour TTL.

Every warning in the cluster, in order. Finds problems in namespaces you were not thinking about.

Events expire after an hour by default, so an absence of warnings is not evidence of health.

API server health per subsystem. A failing check is marked with a minus and names the subsystem.

componentstatuses is deprecated and reports nothing useful. Use this and the kube-system Pod list instead.

The control plane and add-ons. A CoreDNS Pod not Ready explains every resolution failure at once.

Watch restart counts: a number that is climbing is much worse than a high number that has not moved in days.

The scheduler could not place it. The FailedScheduling event accounts for every node and says why each was rejected.

`Insufficient cpu` is about requests, not usage. A node idle in kubectl top can be fully booked.

The requests already committed on a node, which is what the scheduler is actually looking at.

A Pod does not schedule until its PVCs bind. The storage is the cause and the Pod is the symptom.

kubectl describe pvc names the reason, usually a StorageClass that does not exist.

The kubelet cannot set up the network. The CNI is not installed or not running on that node.

On the node: sudo ls /etc/cni/net.d. Empty means the CNI DaemonSet has not run there.

The image could not be pulled. The event message distinguishes not-found from unauthorized from a network failure.

Read the whole message. `not found` is a name or tag error; `unauthorized` needs an imagePullSecret.

The warnings for one namespace, which is where the scheduler's and kubelet's reasons are.

The container exits and the kubelet keeps restarting it with growing backoff.

A report, not an attempt at repair. Kubernetes cannot fix why your process exits.

The previous run's output, which is where the cause is. The current run may show nothing.

The single most useful flag in Kubernetes troubleshooting.

Exit code, reason and signal from the previous run.

Exit 137 is SIGKILL, usually an OOM kill. Exit 1 is the application. Exit 127 is a command not found in the image.

The container exceeded its memory limit and was terminated by the kernel.

Memory is not compressible, so a limit is enforced by killing. Compare limits against kubectl top pods --containers.

A failing readiness probe. The container is up and the Service will not send it traffic.

Readiness removes from endpoints; liveness restarts. Describe shows the probe failures with their HTTP codes.

A liveness probe firing before the application has finished starting.

The fix is a startupProbe, not a longer liveness period, which would also delay real failure detection.

An init container is failing, so the app containers never start.

kubectl logs <pod> -c <init-container>. The app container's logs are empty because it has not run.

The addresses behind a Service. Empty is the most common cause of a Service that does not work.

Empty means selector mismatch, failing readiness, or Pods Pending. Populated means look further along the path.

Timeout means packets dropped, so a policy or firewall. Refused means routing worked and nothing is listening.

They point in opposite directions and are the fastest way to halve the search space.

DNS failed before any connection was attempted.

Under an egress NetworkPolicy this means port 53 to kube-system is not allowed. Otherwise check CoreDNS.

What the Pod was told to use for DNS, including the search domains.

Whether any policy could be involved. Check both the source and destination namespaces.

Also check the CNI's own policy resources; kubectl get netpol shows only networking.k8s.io objects.

Reach the backend bypassing Ingress, Services' external path and NetworkPolicy.

Working here and failing normally localises the problem to the path, not the application.

The access log. A request missing from it never reached the controller.

That absence moves the search in front of the controller: DNS, the load balancer, the node port, or the client.

The conditions behind the word Ready.

For DiskPressure, MemoryPressure, PIDPressure and NetworkUnavailable, False is healthy. Only Ready inverts.

The CNI has not initialised on that node. Pods schedule there and never get an IP.

The condition most often mistaken for a Service or DNS problem.

The kubelet is evicting Pods and refusing new ones.

Usually image or log accumulation. The symptom is Pods vanishing with no obvious cause.

The node has stopped reporting and the controller manager has not flipped it yet.

It flips after node-monitor-grace-period, so a node can be genuinely gone and still read Ready for a short window.

The kubelet's log. Where node-level failures are actually explained.

Run this before anything else on a NotReady node. Config errors, cgroup driver mismatches and CNI failures all surface here.

Whether it is running at all, and the exit status if not.

A kubelet that will not start with swap enabled, or with a cgroup driver that disagrees with the runtime, is the usual pair.

The API server is unreachable, so none of the kubectl-based checks apply.

Switch to the node: sudo crictl ps, sudo journalctl -u kubelet, and the container logs. Also check your kubeconfig points where you think.

Whether the API server container exists and how many times it has exited.

A static Pod's logs without the API server. Often the only way to read why it will not start.

etcd has lost quorum, so the API server cannot read the RBAC objects that would authorise you.

The most misleading symptom in the control plane. Check etcd before reading a single ClusterRoleBinding.

The client certificate was rejected in the TLS handshake, most often expired.

Forbidden means authentication worked and permission did not. Unauthorized means it never got that far.

Ask every etcd member whether it can commit a proposal.

`Error: unhealthy cluster` means at least one member is unhealthy. Two healthy of three is a working cluster; read the per-endpoint lines.

The signature of a member that cannot reach quorum.

Search for this phrase. It is the definitive confirmation of quorum loss.

Every certificate's expiry. Works without a functioning API server.

Leaf certificates last one year. A cluster built and never upgraded stops working on its first birthday.

The rolling update kept old Pods serving, so the Deployment is available and the new ReplicaSet is failing.

Do not check Available alone. Watch for Progressing=False with reason ProgressDeadlineExceeded.

Two ReplicaSets, the new one at 0 ready, is a stalled rollout visible sooner than the condition.

Environment variables are read once at container start, and nothing in the Pod template changed.

kubectl rollout restart, or use a generated ConfigMap whose name carries a content hash so the change is a template change.

Authentication worked; RBAC denied. This is the genuine permissions error.

kubectl auth can-i --list -n <ns> as the affected identity is faster than reading bindings. A new CRD is not covered by existing roles.

Test another identity's permissions without becoming it.

No provisioner answered. Usually a StorageClass that does not exist, or WaitForFirstConsumer with no Pod yet.

The visible symptom is often an unschedulable Pod, one step removed from the cause.

The PV carries node or zone affinity and no suitable node is available.

For new volumes, volumeBindingMode: WaitForFirstConsumer lets the scheduler choose first and avoids it.

FileSystemResizePending. The volume grew and the filesystem has not.

Restart the Pod. Most drivers can only resize a filesystem while the volume is unmounted.

A finalizer is waiting for a controller that may no longer exist.

Clearing finalizers by hand skips whatever external cleanup they existed to do, so check for orphaned resources afterwards.