Python API Pagination and Rate Limits
An API that holds more than a screenful gives it to you in pages, and one that is shared tells you to slow down. Both are loops with a terminating condition you can get wrong. This guide follows the server's own next_page rather than counting, wraps it in a generator whose laziness is a counted fact, and then measures what ignoring Retry-After actually costs: three requests through instead of eight.
APIs and Cloud Guide 27 of 39 Intermediate
- Control nodeUbuntu 26.04 LTS
- Python3.14.4
- requests2.34.2
- paramiko5.0.0
- TimeAbout 18 min
Nothing here is version-specific. yield from needs Python 3.3. The Retry-After header may be either seconds or an HTTP date - this fixture sends seconds, which is what most APIs do.
- pytest9.1.1
- PyYAML6.0.3
- boto3 / botocore1.43.78
| Server Name | IP Address | OS | Roles | CPU | RAM | HDD |
|---|---|---|---|---|---|---|
| RUNNER01 | 192.168.0.27 | Ubuntu 26.04 LTS | Control node - every script in this path runs here | 2 Core | 4 GB | 50 GB |
This guide includes
Use this when the API holds more than a screenful. This matters because a loop that pages by arithmetic is the one that never ends - the server tells you when to stop, and also when to slow down.
- reading the envelope, which tells you about the rest
- letting the server drive the loop rather than arithmetic
- hiding pages behind a generator, so the caller never sees one
- meeting the loop that never ends
- hitting the rate limit, then backing off politely in one place
Before you start
-
One page, and what it tells you about the rest
-
The loop, driven by the server rather than by arithmetic
-
A generator, so the caller never sees a page
-
The loop that never ends
-
The rate limit
-
Hammering it, and then not
-
Paging politely, in one place