CertGrid CertGrid
Concepts·Docker

Docker Swarm Overlay Networks and Routing Mesh

Publish a port on a cluster and every node answers it - including nodes running none of your containers. How the ingress mesh and the service VIP work, proved on three machines.

Swarm and Multi-Host (optional) Guide 43 of 46 Advanced

Tested on the versions above. Captured on a real three-node cluster. Node IDs, IP addresses and join tokens are specific to that cluster - tokens shown here are truncated deliberately and were rotated afterwards. Match the shape of the output, not the values.

Three hosts, because a swarm needs more than one. DOCKER01 is the single manager and Leader; the other two are workers. One manager means no fault tolerance, which several of these guides make a point of - three would tolerate one failure.
Server NameIP AddressOSRolesCPURAMHDD
DOCKER01192.168.0.21Ubuntu 26.04 LTSSwarm Manager (Leader)2 Core4 GB50 GB
DOCKER02192.168.0.22Ubuntu 26.04 LTSSwarm Worker2 Core4 GB50 GB
DOCKER03192.168.0.23Ubuntu 26.04 LTSSwarm Worker2 Core4 GB50 GB

Before you start

  1. Overlay is a network that spans machines

    A bridge network is local to one host. An overlay network is cluster-wide: containers on different machines get addresses on the same subnet and reach each other directly, with the traffic encapsulated between nodes. Note the scope - swarm, not local.

    bash Example session 
    docker network create -d overlay cg-netq3k901yho3i3dc5dwm650uvcrdocker network inspect cg-net --format "driver={{.Driver}} scope={{.Scope}} subnet={{range .IPAM.Config}}{{.Subnet}}{{end}}"driver=overlay scope=swarm subnet=10.0.1.0/24

    Expected resultAn overlay network with cluster scope and its own subnet.

    Success conditionScope is swarm. The network exists on every node that runs a task attached to it - it is created lazily, so it may not appear in network ls on an idle node.

  2. Services find each other by name, across machines

    Attach services to the overlay and each gets a DNS name equal to its service name, resolvable from any task on that network regardless of which machine either is on. This is the single-host behaviour from earlier in the path, extended over the cluster.

    bash Example session 
    docker service create --name cg-api --network cg-net --replicas 3 nginx:alpinedocker service ps cg-api --format "{{.Name}} {{.Node}}"cg-api.1 ahm-docker03cg-api.2 ahm-docker02cg-api.3 ahm-docker01

    Expected resultA successful HTTP fetch using only the service name.

    Verify it worked

    bash 
    docker exec $(docker ps -q -f name=cg-probe) wget -qO- -T5 http://cg-api | head -4<!DOCTYPE html><html><head><title>Welcome to nginx!</title>

    Success conditionThe name resolved and the request was answered by a task that may be on another machine entirely.

  3. One name, one virtual address

    By default the service name resolves to a single virtual IP, not to the task addresses. The VIP is stable for the life of the service and the kernel load-balances behind it, so callers never see tasks appearing and disappearing. If you want the individual addresses - for a client that does its own balancing, or a stateful cluster - tasks. gives them.

    bash Example session 
    docker service inspect cg-api --format "vip={{range .Endpoint.VirtualIPs}}{{.Addr}}{{end}} mode={{.Spec.EndpointSpec.Mode}}"vip=10.0.1.2/24 mode=vipdocker exec $(docker ps -q -f name=cg-probe) nslookup tasks.cg-api | tail -4Address: 10.0.1.5Address: 10.0.1.4

    Expected resultOne VIP for the service, several addresses behind tasks..

    Success conditionYou can explain why the service IP never changes while tasks come and go. --endpoint-mode dnsrr switches to DNS round-robin instead of a VIP.

  4. The routing mesh: every node answers the published port

    This is the part that surprises people. Publishing a port publishes it on EVERY node in the cluster, not just the ones running tasks. Any node that receives the request forwards it to a healthy task somewhere. Nine requests, three sources, three destinations - all answered.

    bash Example session 
    for dst in 21 22 23; do curl -s -m5 -o /dev/null -w "%{http_code} " http://192.168.0.$dst:8080; done200 200 200

    Expected resultEvery node answering on the published port.

    Success conditionAll nine succeed. That is what lets you point a load balancer at any node, or all of them, without tracking placement.

  5. Proof: a node with no tasks still serves

    The strongest demonstration. Drain a node so it runs none of the service's tasks, then request the port from it anyway - it still answers, by forwarding into the mesh.

    bash Example session 
    docker node update --availability drain ahm-docker03docker service ps cg-web --filter desired-state=running --format "{{.Node}}" | sort | uniq -c      3 ahm-docker01      3 ahm-docker02# ahm-docker03 now runs ZERO tasks of this servicecurl -s -m 5 -o /dev/null -w "%{http_code}" http://192.168.0.23:8080200

    Expected result200 from a node hosting none of the containers.

    Success conditionYou have separated "where the container runs" from "where the service answers". Set the node back with --availability active.

  6. ingress, and when to bypass it

    The mesh is implemented by a built-in overlay network called ingress. It costs an extra hop and it hides the client's real IP behind the mesh, which breaks IP-based logging and rate limiting. mode=host publishes on the node's own interface instead - no mesh, only nodes with a task answer, and the real client address arrives intact.

    bash Example session 
    docker network inspect ingress --format "driver={{.Driver}} scope={{.Scope}} subnet={{range .IPAM.Config}}{{.Subnet}}{{end}}"driver=overlay scope=swarm subnet=10.0.0.0/24

    Expected resultThe built-in ingress network, and the alternative publish mode.

    Success conditionYou can choose. Host mode means one task per node on that port, and your load balancer must know which nodes have tasks.

  7. The localhost trap

    Worth knowing before you waste an afternoon. The published port is DNAT'd for IPv4 only. localhost on a modern system resolves to ::1 first, so a request to localhost:8080 is refused while 127.0.0.1:8080 succeeds against the same working service - which reads exactly like a broken cluster.

    bash Example session 
    curl -s -m 5 -o /dev/null -w "localhost:8080 = %{http_code}" http://localhost:8080; echolocalhost:8080 = 000curl -s -m 5 -o /dev/null -w "127.0.0.1:8080 = %{http_code}" http://127.0.0.1:8080; echo127.0.0.1:8080 = 200

    Expected resultThe same service failing by name and succeeding by address.

    Success conditionYou test with an explicit IPv4 address before concluding the mesh is broken. sudo iptables -t nat -L DOCKER-INGRESS -n shows the IPv4-only rule.

Troubleshooting

Official sources

← All Docker guides