kubeadm and node operations cheat sheet

Before init: host prerequisites

sudo swapoff -a

Disable swap for the current boot. Also remove the swap line from /etc/fstab or it returns on reboot.

The kubelet refuses to start with swap on unless failSwapOn is set to false. Disabling it is the normal answer.

Full guide
sudo modprobe br_netfilter overlay

Load the kernel modules the CNI and the runtime need. Persist them in /etc/modules-load.d.

Full guide
sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables

Check the two sysctls Pod networking depends on. Both must be 1.

Set them in /etc/sysctl.d so they survive a reboot; a runtime-only change is the classic post-reboot break.

Full guide
sudo containerd config default | sudo tee /etc/containerd/config.toml

Write a default containerd config, then set SystemdCgroup = true in it.

SystemdCgroup must match the kubelet's cgroup driver. A mismatch produces Pods that start and then die under load.

Full guide
kubeadm config images pull

Pre-pull the control plane images so init is not waiting on the network.

Full guide

Creating a cluster

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Create a single control plane cluster. The CIDR must not overlap your node network.

Full guide
sudo kubeadm init --control-plane-endpoint <host>:<port> --upload-certs

Create a cluster that more control plane nodes can join later.

Both flags must be on the first init. Adding controlPlaneEndpoint afterwards means reissuing certificates and rewriting every kubeconfig.

Full guide
mkdir -p ~/.kube && sudo cp /etc/kubernetes/admin.conf ~/.kube/config && sudo chown $(id -u):$(id -g) ~/.kube/config

Give your user a kubeconfig.

This is a copy. Renewing certificates later updates admin.conf and not this file, which is how kubectl breaks on a cluster whose certificates were renewed.

Full guide
kubectl taint node <name> node-role.kubernetes.io/control-plane:NoSchedule-

Remove the control plane taint so workloads can run on a single-node cluster.

Full guide
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}'

The cluster configuration kubeadm recorded, including controlPlaneEndpoint and the CIDRs.

Full guide

Joining nodes

sudo kubeadm token create --print-join-command

A fresh join command for a worker, with a new 24-hour token.

Tokens expire after 24 hours by default. A join failing on a cluster built yesterday is usually this.

Full guide
sudo kubeadm init phase upload-certs --upload-certs

Re-upload the control plane certificates and print a new certificate key.

Uploaded certificates are deleted two hours after init, so a control plane join later needs this first.

Full guide
sudo kubeadm join <endpoint> --token <t> --discovery-token-ca-cert-hash sha256:<h>

Join as a worker.

Full guide
sudo kubeadm join <endpoint> --token <t> --discovery-token-ca-cert-hash sha256:<h> --control-plane --certificate-key <k>

Join as an additional control plane, which also adds an etcd member.

One node at a time, and only while the cluster is healthy: the join is itself a write to etcd, and going from one member to two temporarily tolerates no failures.

Full guide
kubeadm token list

Existing bootstrap tokens and their expiry.

Full guide

Upgrading

sudo kubeadm upgrade plan

What versions you can upgrade to and what it will change. Reads only.

Full guide
sudo kubeadm upgrade apply v1.36.4

Upgrade the control plane on the first node. Also renews certificates.

A cluster upgraded at least once a year never meets certificate expiry, because this renews them as a side effect.

Full guide
sudo kubeadm upgrade node

Upgrade the local component configuration on every other node, control plane or worker.

Full guide
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=<version> && sudo apt-mark hold kubeadm

The package dance: the hold is what stops an unplanned upgrade from a routine apt upgrade.

Order matters. Upgrade kubeadm, run the upgrade, then upgrade kubelet and kubectl and restart the kubelet.

Full guide
sudo systemctl daemon-reload && sudo systemctl restart kubelet

Pick up a new kubelet binary or unit configuration.

Full guide

Certificates

sudo kubeadm certs check-expiration

Every managed certificate with its expiry, and the CAs separately.

Leaf certificates last one year; the CAs last ten. EXTERNALLY MANAGED yes means kubeadm cannot renew it.

Full guide
sudo kubeadm certs renew all

Renew every leaf certificate, including the ones embedded in /etc/kubernetes/*.conf.

Works without a functioning API server, because it signs locally from the CA keys. That is what makes it a recovery tool.

Full guide
sudo kubeadm certs renew apiserver

Renew one certificate. Used after adding a name to apiServerCertSANs.

Full guide
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates -ext subjectAltName

Read validity and the SAN list straight off the certificate.

A client connecting by a name absent from the SANs fails verification, and the error blames the certificate rather than the address.

Full guide
echo | openssl s_client -connect 127.0.0.1:6443 | openssl x509 -noout -dates

The certificate actually being served, as opposed to the one on disk.

The only way to confirm a component restarted and picked up a renewed certificate.

Full guide
kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -dates

The expiry of the client certificate in your own kubeconfig.

Run this on any Unauthorized error before looking at RBAC.

Full guide

etcd

sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table

The etcd members, their peer and client addresses, and whether any is a learner.

etcdctl is not shipped with kubeadm. Install the release matching the etcd image the cluster runs.

Full guide
sudo etcdctl ... endpoint health --cluster

Ask every member whether it can commit a proposal.

`Error: unhealthy cluster` means at least one member is unhealthy, not that quorum is lost. Read the per-endpoint lines.

Full guide
sudo etcdctl ... endpoint status --cluster -w table

Per-member version, database size and which one is the leader.

Full guide
sudo etcdctl ... snapshot save /var/backups/etcd-$(date +%F).db

Take a snapshot. The only thing that protects the cluster's data.

A snapshot you have not restored from is a hypothesis. Rehearse the restore.

Full guide
sudo etcdctl snapshot restore <file> --data-dir /var/lib/etcd-new Whole-host

Restore into a fresh data directory, then point the etcd static Pod at it.

Never restore over a live data directory, and stop the API server first.

Full guide
sudo etcdctl ... member remove <id> Caution

Remove a dead member before joining its replacement.

Skipping this leaves the cluster counting a member that will never vote, which quietly reduces how many failures it tolerates.

Full guide

On the node, when kubectl will not work

sudo systemctl status kubelet

Whether the kubelet is running. Everything on a node depends on it.

Full guide
sudo journalctl -u kubelet -n 50 --no-pager

The kubelet's own log, which is where node-level failures are explained.

The first place to look when a node is NotReady or the control plane is unreachable.

Full guide
sudo crictl ps

Running containers, from the runtime rather than the API. Works with the API server down.

Full guide
sudo crictl ps -a --name kube-apiserver

Including exited containers, which is how you find one that is crash-looping.

Full guide
sudo crictl logs <container-id>

A container's logs without the API server.

For a static Pod that never becomes healthy, this is the only way to read why.

Full guide
sudo ls /etc/kubernetes/manifests/

The static Pod manifests: etcd, kube-apiserver, kube-controller-manager, kube-scheduler.

Full guide
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/ && sleep 20 && sudo mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/ Caution

Restart one control plane component: the kubelet stops the Pod when the file leaves and starts it when it returns.

The precise way to restart after renewing certificates or changing a flag, and to simulate a component failing.

Full guide
sudo ls /etc/cni/net.d/ /opt/cni/bin/

The CNI config the kubelet reads and the plugin binaries it executes.

Empty /etc/cni/net.d is why Pods sit in ContainerCreating with a CNI error. It means the CNI DaemonSet has not run here.

Full guide
sudo kubeadm reset -f Whole-host

Undo kubeadm on this node: stop the components, remove the manifests and the etcd data.

Does not clean /etc/cni/net.d, the CNI interfaces or iptables rules. Remove those by hand or the rebuild inherits them.

Full guide
kubectl delete node <name> Destructive

Remove the Node object after resetting the machine.

On a control plane node, also remove its etcd member.

Full guide