CertGrid CertGrid
Hands-on Lab·Python Automation for IT

Python API Pagination and Rate Limits

An API that holds more than a screenful gives it to you in pages, and one that is shared tells you to slow down. Both are loops with a terminating condition you can get wrong. This guide follows the server's own `next_page` rather than counting, wraps it in a generator whose laziness is a counted fact, and then measures what ignoring `Retry-After` actually costs: three requests through instead of eight.

APIs and Cloud Guide 27 of 39 Intermediate

Written against the versions above. Nothing here is version-specific. `yield from` needs Python 3.3. The `Retry-After` header may be either seconds or an HTTP date - this fixture sends seconds, which is what most APIs do.

Everything on this page runs on the control node. Any machine with Python 3 will do.
Server NameIP AddressOSRolesCPURAMHDD
RUNNER01192.168.0.27Ubuntu 26.04 LTSControl node - every script in this path runs here2 Core4 GB50 GB

Before you start

  1. One page, and what it tells you about the rest

  2. The loop, driven by the server rather than by arithmetic

  3. A generator, so the caller never sees a page

  4. The loop that never ends

  5. The rate limit

  6. Hammering it, and then not

  7. Paging politely, in one place

Official sources

← All Python Automation guides