Resetting and Rejoining a Node with kubeadm

Installation and Cluster Builds / Guide 13 of 103 / Intermediate

• Kubernetes1.36.4
• Cluster2 nodes
• Runtimecontainerd 2.2.6
• CNICalico v3.32.1
• TimeAbout 30 min

Container and sandbox IDs, timings and package versions are specific to this run.

• Calicov3.32.1

This guide includes

Use this when a node has to leave a cluster, come back, or be handed on as a clean machine. This matters because removing a node is two separate pieces of state, and the package purge people reach for does nothing at all while the packages are held.

• removing a node properly - drain, delete the Node object, then reset the machine
• reading what kubeadm reset reports, and expecting noisy StopPodSandbox errors where a CNI was installed
• rejoining with a freshly minted token rather than the dead one from kubeadm init
• taking the machine back to bare Ubuntu, including the CNI directories reset leaves behind
• unholding the packages first, because the purge otherwise does nothing, then rebooting - Calico's interfaces outlive them

Before you start

• A cluster with at least one worker you are willing to disturb.
• sudo on the node and kubectl access to the control plane.
• If the node runs anything you care about, somewhere else for it to go. Removing the only worker leaves its pods Pending.
• Know which you want: taking a node out of a cluster is not the same as taking a machine back to bare Ubuntu. The first is two commands, the second is closer to eight.

1. 1

   Removing a node takes two commands, not one

2. 2

   Reset the machine, and read what it tells you

3. 3

   Expect noisy errors during reset on a node that had a CNI

4. 4

   Rejoin with a fresh token

5. 5

   Going further: back to bare Ubuntu

6. 6

   The package purge that silently does nothing

7. 7

   Reboot, because interfaces outlive the packages

Official sources

• kubeadm reset
• kubeadm join
• kubeadm token
• Safely drain a node