Python Retries, Timeouts and Backoff
requests has no default timeout and no retries, so an unattended script gets both wrong by default: it can hang for ever, and it gives up on a failure that would have succeeded a second later. This guide separates connect from read timeouts, builds backoff with jitter, hands the job to urllib3's Retry, and then draws the line - because retrying a 404 is pointless and retrying a POST can create two of something.
APIs and Cloud Guide 28 of 39 Advanced
- Control nodeUbuntu 26.04 LTS
- Python3.14.4
- requests2.34.2
- paramiko5.0.0
- TimeAbout 22 min
urllib3.util.retry.Retry uses allowed_methods; the old name method_whitelist was removed in urllib3 2.0. respect_retry_after_header defaults to True.
- pytest9.1.1
- PyYAML6.0.3
- boto3 / botocore1.43.78
| Server Name | IP Address | OS | Roles | CPU | RAM | HDD |
|---|---|---|---|---|---|---|
| RUNNER01 | 192.168.0.27 | Ubuntu 26.04 LTS | Control node - every script in this path runs here | 2 Core | 4 GB | 50 GB |
This guide includes
Use this for anything that calls a network service unattended. This matters because requests has no default timeout at all - so one hung server holds your script for as long as it likes.
- calling a slow endpoint with and without a timeout, and splitting connect from read
- seeing what is wrong with the naive retry
- retrying with backoff and jitter, and reading the delays that schedule produces
- mounting a
Retryon the Session and letting urllib3 do it - deciding what must not be retried
Before you start
-
A slow endpoint, with and without a timeout
-
Connect and read are two different waits
-
Something worth retrying
-
The naive retry, and what is wrong with it
-
Backoff, with jitter
-
The delays that schedule produces
-
Or let urllib3 do it
-
What must not be retried