Highly Available Control Planes and etcd Quorum

Cluster Architecture and Configuration / Guide 21 of 103 / Advanced

• Kubernetes1.36.4
• Cluster3 nodes
• Runtimecontainerd 2.2.6
• CNICalico v3.32.1
• TimeAbout 75 min

Stacked etcd, the default kubeadm HA topology. An external etcd cluster is the other option and changes the join procedure but not the quorum arithmetic.

• etcd3.6.8
• keepalived + haproxydistribution packages

This guide includes

Use this when the control plane has to survive losing a node. This matters because HA is a quorum problem before it is a Kubernetes problem - three members tolerate one failure, and the second failure stops writes altogether.

• understanding why an HA control plane is three nodes rather than two
• standing up keepalived and haproxy so the endpoint exists before the first kubeadm init
• initialising with --control-plane-endpoint and --upload-certs, then joining two more control planes
• stopping keepalived, watching the VIP move, and seeing kubectl not notice
• removing one etcd member and then a second - losing quorum, and reading the Forbidden that follows

Before you start

• The kubeadm cluster install guide, since this is that procedure with one flag added.
• The etcd guides, because quorum is the whole subject.
• Three machines with kubeadm, kubelet and a container runtime installed, and a free address on their subnet for the VIP.

1. 1

   Three, not two, and why

2. 2

   The endpoint has to exist before the first init

3. 3

   init with --control-plane-endpoint and --upload-certs

4. 4

   Joining the other two

5. 5

   What three control planes actually look like

6. 6

   The VIP moves, and kubectl does not notice

7. 7

   Losing one member, and the health-check window

8. 8

   Losing two: quorum gone, and it reports Forbidden

Official sources

• Creating Highly Available clusters with kubeadm
• Options for Highly Available topology
• Operating etcd clusters for Kubernetes
• etcd FAQ on cluster size
• Set up a load balancer for kube-apiserver