CertGrid CertGrid
Best Practices·Python Automation for IT

Python Multi-Host Failure Handling

A loop over an inventory has one host that is down, one whose name does not resolve and one where the command itself fails - and the obvious for loop stops at the first of them, having done a third of the work and reported nothing. This guide collects a result per host, distinguishes failures that are worth retrying from those that are not, runs them concurrently for a measured 4.1x, and exits with a code that says whether some or all of it worked.

Remote Hosts and CI/CD Guide 34 of 39 Advanced

concurrent.futures.ThreadPoolExecutor is standard library and unchanged in the relevant respects. Threads are right here because every worker is blocked on a socket - the GIL is not in the way.

One control node and two managed hosts. The targets keep the hostnames the RHCSA path gave them - they are ordinary Linux boxes reachable over SSH, and nothing here is Red Hat specific.
Server NameIP AddressOSRolesCPURAMHDD
RUNNER01192.168.0.27Ubuntu 26.04 LTSControl node - every script in this path runs here2 Core4 GB50 GB
RHCSA-A01192.168.0.31RHEL 10.0Managed host - reached over SSH from the control node2 Core4 GB50 GB
RHCSA-B01192.168.0.33RHEL 10.0Second managed host - so an inventory has more than one row2 Core4 GB50 GB

This guide includes

Use this for any loop over an inventory. This matters because the three ways a host fails are not the same failure - and a loop that stops at the first one tells you nothing about the rest.

Before you start

  1. The loop that stops at the first problem

  2. So catch per host, and keep a result for each

  3. The three failures are not the same failure

  4. Sequentially, then in parallel

  5. And the worker count that matters

  6. An exception inside a worker, and where it surfaces

  7. What the script should exit with

Official sources

← All Python Automation guides