CertGrid CertGrid
Concepts·Python Automation for IT

Python Standard Library for Automation

Most automation needs no dependency at all: paths, CSV, JSON, INI, TOML, SQLite, regex, subprocess, logging, argparse and an HTTP client all ship with the interpreter. Knowing that matters because every dependency is something to install on a host you may not control. This guide shows what is free, what genuinely earns a dependency, and the three different places `import requests` can be satisfied from on one machine - with three different versions.

Start Here Guide 3 of 39 Beginner

Written against the versions above. `tomllib` arrived in **Python 3.11** and is read-only, which is the one version caveat in this guide. Everything else here has been in the standard library for many releases.

Everything on this page runs on the control node. Any machine with Python 3 will do.
Server NameIP AddressOSRolesCPURAMHDD
RUNNER01192.168.0.27Ubuntu 26.04 LTSControl node - every script in this path runs here2 Core4 GB50 GB

Before you start

  1. Eighteen jobs, no installs

    Each line is a job this track needs doing and the standard library module that does it. Nothing here was installed.

    bash Example session 
    cat > ~/auto/std.py <<'PY'jobs = [    ("paths and directories",   "pathlib"),    ("run another program",     "subprocess"),    ("CSV",                     "csv"),    ("JSON",                    "json"),    ("INI",                     "configparser"),    ("TOML (read only)",        "tomllib"),    ("SQLite",                  "sqlite3"),    ("regular expressions",     "re"),    ("HTTP client",             "urllib.request"),    ("command-line arguments",  "argparse"),    ("logging",                 "logging"),    ("temp files",              "tempfile"),    ("copy, move, archive",     "shutil"),    ("test doubles",            "unittest.mock"),    ("a local HTTP server",     "http.server"),    ("dates and times",         "datetime"),    ("quoting for the shell",   "shlex"),    ("environment",             "os"),] for job, module in jobs:    __import__(module)    print("%-26s %s" % (job, module))PYpython3 ~/auto/std.pypaths and directories      pathlibrun another program        subprocessCSV                        csvJSON                       jsonINI                        configparserTOML (read only)           tomllibSQLite                     sqlite3regular expressions        reHTTP client                urllib.requestcommand-line arguments     argparselogging                    loggingtemp files                 tempfilecopy, move, archive        shutiltest doubles               unittest.mocka local HTTP server        http.serverdates and times            datetimequoting for the shell      shlexenvironment                os

    Expected resultEighteen jobs paired with their modules, no import errors.

    Success conditionYou know what is available on any Python 3 machine.

  2. The same GET, twice

    One HTTP request against the lab API, written both ways. The standard library version is four lines and the requests version is two.

    bash Example session 
    cat > ~/auto/with_urllib.py <<'PY'import jsonimport urllib.request with urllib.request.urlopen("http://127.0.0.1:8000/hosts", timeout=5) as resp:    data = json.loads(resp.read()) print("urllib  ", resp.status, "->", [h["name"] for h in data["hosts"]])PYpython3 ~/auto/with_urllib.pyurllib   200 -> ['web01', 'web02']cat > ~/auto/with_requests.py <<'PY'import requests resp = requests.get("http://127.0.0.1:8000/hosts", timeout=5) print("requests", resp.status_code, "->", [h["name"] for h in resp.json()["hosts"]])PYpython3 ~/auto/with_requests.pyrequests 200 -> ['web01', 'web02']

    Expected result200 and the same two host names from both.

    Success conditionYou can make an HTTP call with no dependency at all.

  3. Three places a module can come from

    This is the step that matters. The same script, run twice on the same machine by two different interpreters, printing where each import was satisfied from.

    Read the two outputs against each other.

    bash Example session 
    cat > ~/auto/whence.py <<'PY'import importlibimport sys print("interpreter:", sys.executable)for name in ("json", "requests", "yaml", "paramiko", "pytest"):    try:        mod = importlib.import_module(name)    except ModuleNotFoundError:        print("  %-9s NOT AVAILABLE" % name)        continue    where = getattr(mod, "__file__", None) or "built in"    version = getattr(mod, "__version__", "-")    print("  %-9s %-8s %s" % (name, version, where))PYpython3 ~/auto/whence.pyinterpreter: /usr/bin/python3  json      2.0.9    /usr/lib/python3.14/json/__init__.py  requests  2.32.5   /usr/lib/python3/dist-packages/requests/__init__.py  yaml      6.0.3    /usr/lib/python3/dist-packages/yaml/__init__.py  paramiko  NOT AVAILABLE  pytest    NOT AVAILABLE~/autoenv/bin/python ~/auto/whence.pyinterpreter: /home/sysadmin/autoenv/bin/python  json      2.0.9    /usr/lib/python3.14/json/__init__.py  requests  2.34.2   /home/sysadmin/autoenv/lib/python3.14/site-packages/requests/__init__.py  yaml      6.0.3    /home/sysadmin/autoenv/lib/python3.14/site-packages/yaml/__init__.py  paramiko  5.0.0    /home/sysadmin/autoenv/lib/python3.14/site-packages/paramiko/__init__.py  pytest    9.1.1    /home/sysadmin/autoenv/lib/python3.14/site-packages/pytest/__init__.py

    Expected resultOn the system Python: requests 2.32.5 from dist-packages, and paramiko and pytest not available. In the venv: requests 2.34.2 from site-packages, and all four present.

    Success conditionYou can say where any import on your machine actually came from.

  4. The apt packages behind the system copies

    Not a mystery - two ordinary Debian packages, visible to dpkg.

    bash Example session 
    dpkg -l | grep -E "python3-(requests|yaml) " | awk '{print $2, $3}'python3-requests 2.32.5+dfsg-1ubuntu1python3-yaml 6.0.3-1build1

    Expected resultpython3-requests 2.32.5+dfsg-1ubuntu1 and python3-yaml 6.0.3-1build1.

    Success conditionYou know why the system Python has libraries you never installed.

  5. Which interpreter runs the script decides what it can do

    paramiko is the clean test, because unlike requests it is not an apt package on this machine - so it exists in the venv and nowhere else.

    The same script, run by each interpreter.

    bash Example session 
    cat > ~/auto/needs_paramiko.py <<'PY'import paramiko print("paramiko", paramiko.__version__, "- this script can reach a host over SSH")PYpython3 ~/auto/needs_paramiko.pyTraceback (most recent call last):  File "/home/sysadmin/auto/needs_paramiko.py", line 1, in <module>    import paramikoModuleNotFoundError: No module named 'paramiko'[exit 1]~/autoenv/bin/python ~/auto/needs_paramiko.pyparamiko 5.0.0 - this script can reach a host over SSH

    Expected resultModuleNotFoundError: No module named 'paramiko' from the system Python; paramiko 5.0.0 from the venv.

    Success conditionYou can diagnose a ModuleNotFoundError for a library you know you installed.

  6. One standard library module that only does half the job

    tomllib reads TOML and cannot write it - deliberately, and it is the one asymmetry in this guide worth remembering.

    bash Example session 
    python3 -c "import tomllib; print('loads:', tomllib.loads('port = 8000')); print('has dumps:', hasattr(tomllib, 'dumps'))"loads: {'port': 8000}has dumps: False

    Expected result{'port': 8000} parsed, and has dumps: False.

    Success conditionYou know the one standard library format that is read-only.

  7. The rule to take into the rest of the path

    > Reach for the standard library first. Add a dependency when it does a job the standard library does not do at all, not when it does a job more prettily.

    By that test, this path adds exactly four:

    | Dependency | The job the standard library cannot do | |---|---| | PyYAML | there is no YAML parser in the standard library, at all | | Paramiko | there is no SSH client either | | pytest | unittest works, but fixtures and plain assert are worth the install | | requests | sessions, retries and sane error handling that urllib makes genuinely hard |

    The first two are unarguable. The second two are judgement calls, and both guides say so where they come up.

    That is the orientation track. guide 4 starts domain 1.

    bash Example session 
    rm -f ~/auto/std.py ~/auto/with_urllib.py ~/auto/with_requests.py ~/auto/whence.py ~/auto/needs_paramiko.py && ls -A ~/autolabapi.loglabapi.pidlabapi.py

    Expected resultThe scratch directory back to the lab's API fixture.

    Success conditionYou can decide whether a script needs a dependency.

Troubleshooting

Official sources

← All Python Automation guides