CertGrid CertGrid
Best Practices·Python Automation for IT

Python Multi-Host Failure Handling

A loop over an inventory has one host that is down, one whose name does not resolve and one where the command itself fails - and the obvious `for` loop stops at the first of them, having done a third of the work and reported nothing. This guide collects a result per host, distinguishes failures that are worth retrying from those that are not, runs them concurrently for a measured 4.1x, and exits with a code that says whether some or all of it worked.

Remote Hosts and CI/CD Guide 34 of 39 Advanced

Written against the versions above. `concurrent.futures.ThreadPoolExecutor` is standard library and unchanged in the relevant respects. Threads are right here because every worker is blocked on a socket - the GIL is not in the way.

One control node and two managed hosts. The targets keep the hostnames the RHCSA path gave them - they are ordinary Linux boxes reachable over SSH, and nothing here is Red Hat specific.
Server NameIP AddressOSRolesCPURAMHDD
RUNNER01192.168.0.27Ubuntu 26.04 LTSControl node - every script in this path runs here2 Core4 GB50 GB
RHCSA-A01192.168.0.31RHEL 10.0Managed host - reached over SSH from the control node2 Core4 GB50 GB
RHCSA-B01192.168.0.33RHEL 10.0Second managed host - so an inventory has more than one row2 Core4 GB50 GB

Before you start

  1. The loop that stops at the first problem

  2. So catch per host, and keep a result for each

  3. The three failures are not the same failure

  4. Sequentially, then in parallel

  5. And the worker count that matters

  6. An exception inside a worker, and where it surfaces

  7. What the script should exit with

Official sources

← All Python Automation guides