Hands-on Lab·Certified Kubernetes Administrator
Troubleshooting a Stalled Rollout
A Deployment that stops half way and keeps serving traffic while it does. This reads both conditions, shows the two ReplicaSets that make it safe, and undoes it.
Troubleshooting Guide 86 of 103 Intermediate
- Kubernetes1.36.4
- Cluster4 nodes
- Runtimecontainerd 2.2.6
- CNICalico v3.32.1
- TimeAbout 16 min
progressDeadlineSeconds is set to 30 here so the failure is watchable. The default is 600, which is ten minutes of looking like nothing is wrong.
| Server Name | IP Address | OS | Roles | CPU | RAM | HDD |
|---|---|---|---|---|---|---|
| CKA1001 | 192.168.0.175 | Ubuntu 26.04 LTS | Control Plane Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE01 | 192.168.0.176 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE02 | 192.168.0.177 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE03 | 192.168.0.178 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
This guide includes
Use this when a Deployment has stopped half way and is still serving traffic. This matters because a stalled rollout is not an outage - two ReplicaSets exist at once, and the old one is what keeps the site up while you decide what to do.
- setting
progressDeadlineSecondsshort enough that the failure is watchable at all - rolling out an image that cannot start, and reading READY against UP-TO-DATE
- reading the two Deployment conditions together, which is what says stalled rather than failed
- finding both ReplicaSets, which is the mechanism behind everything above
- undoing it by re-pointing the Deployment at the previous ReplicaSet
Before you start
- guide 26 - the mechanism, before this covers it failing.
-
A healthy Deployment, with a short deadline so this is watchable
-
Roll out an image that cannot start
-
What the Deployment says about it
-
The two ReplicaSets
-
Undo it