CertGrid CertGrid
Hands-on Lab·Docker

Docker Swarm Node Maintenance and Failover

Take a machine out of service without dropping a request. Drain it, watch six replicas rebalance across the remaining two nodes, confirm it still answers on the published port, then bring it back.

Swarm and Multi-Host (optional) Guide 45 of 46 Advanced

Tested on the versions above. Captured on a real three-node cluster. Node names, IDs and addresses are specific to it - match the shape of the output, not the values.

Three hosts, because a swarm needs more than one. DOCKER01 is the single manager and Leader; the other two are workers. One manager means no fault tolerance, which several of these guides make a point of - three would tolerate one failure.
Server NameIP AddressOSRolesCPURAMHDD
DOCKER01192.168.0.21Ubuntu 26.04 LTSSwarm Manager (Leader)2 Core4 GB50 GB
DOCKER02192.168.0.22Ubuntu 26.04 LTSSwarm Worker2 Core4 GB50 GB
DOCKER03192.168.0.23Ubuntu 26.04 LTSSwarm Worker2 Core4 GB50 GB

Before you start

  1. Status and availability are different things

    STATUS is what the cluster observes: is this node reachable. AVAILABILITY is what you have decided: may it be given work. You control the second one. Conflating them is why people reach for stopping the daemon when they only wanted to stop scheduling.

    bash Example session 
    docker node lsID                            HOSTNAME       STATUS    AVAILABILITY   MANAGER STATUSfr31hknhlxd5a9s4324qwqqy1 *   ahm-docker01   Ready     Active         Leadery9vvv2k6w5ur7kjodwnro59ej     ahm-docker02   Ready     Activevqacq9wzoqon1nkz9bd9m0l3j     ahm-docker03   Ready     Active

    Expected resultTwo separate columns, both healthy.

    Success conditionYou can name the three availability values: active takes new work, pause keeps existing tasks but accepts no new ones, drain evicts everything.

  2. The starting spread

    Six replicas, two per node. This is the state we are about to disturb deliberately.

    bash Example session 
    docker service ps cg-web --filter desired-state=running --format "{{.Node}}" | sort | uniq -c      2 ahm-docker01      2 ahm-docker02      2 ahm-docker03

    Expected resultAn even spread across three nodes.

    Success conditionYou have a baseline to compare against.

  3. Drain the node

    One command evicts every task from a node and refuses it new ones. The tasks are not moved - each is stopped and a REPLACEMENT is scheduled elsewhere, which is why a stateless service handles this transparently and a stateful one needs its data on shared or replicated storage.

    bash Example session 
    docker node update --availability drain ahm-docker03ahm-docker03docker node lsvqacq9wzoqon1nkz9bd9m0l3j     ahm-docker03   Ready     Drain

    Expected resultAvailability Drain while STATUS stays Ready - the node is healthy, just not accepting work.

    Success conditionReady and Drain together. The machine is up; the scheduler is simply ignoring it.

  4. Watch the tasks move

    All six replicas are now on two nodes, and the history shows what happened: the task on the drained node was shut down and a new one started elsewhere. The service never dropped below its declared replica count for long.

    bash Example session 
    docker service ps cg-web --filter desired-state=running --format "{{.Node}}" | sort | uniq -c      3 ahm-docker01      3 ahm-docker02docker service ps cg-web --format "table {{.Name}}\t{{.Node}}\t{{.DesiredState}}\t{{.CurrentState}}" | head -5NAME           NODE           DESIRED STATE   CURRENT STATEcg-web.1       ahm-docker02   Running         Running 6 seconds ago \_ cg-web.1   ahm-docker03   Shutdown        Shutdown 6 seconds ago

    Expected resultSix tasks on two nodes, with shutdown history on the drained one.

    Success conditionReplica count is intact. Note the cluster did NOT rebalance back onto the remaining nodes evenly by moving anything - it only placed the replacements.

  5. The drained node still serves

    It is running none of the service's containers, yet the published port still answers - the routing mesh forwards to a task elsewhere. This is why draining a node for maintenance does not require touching your load balancer.

    bash Example session 
    curl -s -m 5 -o /dev/null -w "%{http_code}" http://192.168.0.23:8080; echo200

    Expected result200 from a node with no tasks.

    Success conditionTraffic is unaffected. You can now patch, reboot or reconfigure that machine at leisure.

  6. Bring it back

    Setting availability back to active makes the node eligible again. It does NOT immediately reclaim tasks - the cluster does not rebalance a healthy service just because capacity appeared. New tasks will land there as the service changes.

    bash Example session 
    docker node update --availability active ahm-docker03ahm-docker03docker node ls --format "{{.Hostname}} {{.Availability}}"ahm-docker01 Activeahm-docker02 Activeahm-docker03 Active

    Expected resultAll three active again.

    Success conditionThe node is eligible. To spread work back over it now, docker service update --force NAME redeploys tasks and lets the scheduler place them afresh.

  7. Removing a node for good

    Draining is temporary. Retiring a machine is a sequence, and doing it in the wrong order leaves a ghost node in node ls that you then have to force-remove.

    bash 
    # 1. stop giving it work and let tasks move offdocker node update --availability drain ahm-docker03# 2. on the node itself, leave the clusterdocker swarm leave# 3. back on a manager, remove the now-Down entrydocker node rm ahm-docker03# a MANAGER must be demoted first: docker node demote NAME

    Expected resultA clean removal in three steps.

    Success conditiondocker node ls no longer lists it. Removing a manager without demoting it first risks losing quorum.

Troubleshooting

Official sources

← All Docker guides