CertGrid CertGrid
Hands-on Lab·Python Automation for IT

Python Configuration and Environment Variables

A script with a hostname in it works on one host. Settings belong outside the code, and in practice they come from four places at once - a default in the code, a configuration file, an environment variable, and the command line - with a precedence order that has to be deliberate. This guide builds that layering and prints which layer won, then deals with the one setting that must never be in the file: a credential.

Scripting Foundations Guide 7 of 39 Beginner

Written against the versions above. `configparser` and `os.environ` are unchanged across Python 3. `tomllib` for reading TOML arrived in **3.11** and is covered in {{guide:yaml-ini-and-toml}}; this guide uses INI, which needs nothing.

Everything on this page runs on the control node. Any machine with Python 3 will do.
Server NameIP AddressOSRolesCPURAMHDD
RUNNER01192.168.0.27Ubuntu 26.04 LTSControl node - every script in this path runs here2 Core4 GB50 GB

Before you start

  1. The environment a script can already see

    os.environ is a mapping of the process's environment variables. It behaves like a dictionary, so a missing key raises KeyError and .get() returns None or a default.

    bash Example session 
    cat > ~/auto/env.py <<'PY'import os print("HOME        ", os.environ["HOME"])print("USER        ", os.environ.get("USER"))print("MISSING     ", os.environ.get("NO_SUCH_VAR"))print("with default", os.environ.get("NO_SUCH_VAR", "a fallback"))print("getenv      ", os.getenv("NO_SUCH_VAR", "same thing"))print("how many    ", len(os.environ), "variables")PYpython3 ~/auto/env.pyHOME         /home/sysadminUSER         sysadminMISSING      Nonewith default a fallbackgetenv       same thinghow many     16 variables

    Expected resultHOME and USER, then None for a variable that does not exist, then two fallbacks.

    Success conditionYou can read a setting from the environment.

  2. A missing variable, two ways

    Subscripting raises. Which is sometimes exactly what you want - a script that cannot work without a credential should stop immediately - but the traceback is not a good way to say so.

    bash Example session 
    cat > ~/auto/envstrict.py <<'PY'import os print(os.environ["API_TOKEN"])PYpython3 ~/auto/envstrict.pyTraceback (most recent call last):  File "/home/sysadmin/auto/envstrict.py", line 3, in <module>    print(os.environ["API_TOKEN"])          ~~~~~~~~~~^^^^^^^^^^^^^  File "<frozen os>", line 709, in __getitem__KeyError: 'API_TOKEN'[exit 1]API_TOKEN=s3cret python3 ~/auto/envstrict.pys3cret

    Expected resultKeyError: 'API_TOKEN', then the value once it is set.

    Success conditionYou can see the difference between required and optional settings.

  3. Everything from the environment is a string

    Including numbers, and including things that look like booleans. The bool("false") line is the trap.

    bash Example session 
    cat > ~/auto/envtypes.py <<'PY'import os raw_timeout = os.environ.get("TIMEOUT", "30")raw_debug = os.environ.get("DEBUG", "") print("TIMEOUT raw  ", repr(raw_timeout), type(raw_timeout).__name__)print("TIMEOUT as int", int(raw_timeout) + 1)print("DEBUG raw    ", repr(raw_debug))print("bool(DEBUG)  ", bool(raw_debug), "<- careful")print("the right test", raw_debug.lower() in ("1", "true", "yes", "on"))PYTIMEOUT=5 DEBUG=false python3 ~/auto/envtypes.pyTIMEOUT raw   '5' strTIMEOUT as int 6DEBUG raw     'false'bool(DEBUG)   True <- carefulthe right test FalseTIMEOUT=5 DEBUG=1 python3 ~/auto/envtypes.pyTIMEOUT raw   '5' strTIMEOUT as int 6DEBUG raw     '1'bool(DEBUG)   True <- carefulthe right test True

    Expected result'5' as a string then 6 as an int; and bool("false") reporting True.

    Success conditionYou convert every value you read, deliberately.

  4. An INI file, with configparser

    The standard library's configuration format, and it needs no dependency. Sections in brackets, key = value inside, and a [DEFAULT] section every other section inherits from.

    bash Example session 
    cat > ~/auto/app.ini <<'INI'[DEFAULT]timeout = 30retries = 3 [web]host = web01.example.comport = 8080tls = yes [db]host = db01.example.comport = 5432retries = 5INIcat > ~/auto/readini.py <<'PY'import configparser cfg = configparser.ConfigParser()cfg.read("app.ini") print("sections      ", cfg.sections())print("web host      ", cfg["web"]["host"])print("web port int  ", cfg.getint("web", "port"))print("web tls bool  ", cfg.getboolean("web", "tls"))print("web timeout   ", cfg.getint("web", "timeout"), "<- inherited from [DEFAULT]")print("db retries    ", cfg.getint("db", "retries"), "<- overrides [DEFAULT]")print("fallback      ", cfg.get("web", "nosuch", fallback="not set"))PYcd ~/auto && python3 readini.pysections       ['web', 'db']web host       web01.example.comweb port int   8080web tls bool   Trueweb timeout    30 <- inherited from [DEFAULT]db retries     5 <- overrides [DEFAULT]fallback       not set

    Expected resultThe sections, typed values from getint and getboolean, timeout inherited from [DEFAULT], and retries overridden in [db].

    Success conditionYou can read a configuration file with no dependency.

  5. And what it raises without a fallback

    A specific exception with a specific name, which is what you want to catch.

    bash Example session 
    cd ~/auto && python3 -c "import configparser; c = configparser.ConfigParser(); c.read('app.ini'); print(c['web']['nosuch'])"Traceback (most recent call last):  File "<string>", line 1, in <module>    import configparser; c = configparser.ConfigParser(); c.read('app.ini'); print(c['web']['nosuch'])                                                                                   ~~~~~~~~^^^^^^^^^^  File "/usr/lib/python3.14/configparser.py", line 1306, in __getitem__    raise KeyError(key)KeyError: 'nosuch'[exit 1]

    Expected resultconfigparser.NoOptionError naming the section and the option.

    Success conditionYou can tell a missing setting from a malformed one.

  6. The layering a real script uses

    Four sources, in increasing precedence: defaults in the code, then a file, then the environment, then the command line. Each one overrides the one before.

    layered.py records where each value came from, so the precedence is visible rather than asserted. First run: no file, no environment, no arguments.

    bash Example session 
    cat > ~/auto/layered.py <<'PY'"""Defaults, then a file, then the environment, then the command line."""import argparseimport configparserimport osfrom pathlib import Path DEFAULTS = {"host": "localhost", "port": "8000", "timeout": "30"}  def load(path, argv=None):    settings = dict(DEFAULTS)    source = {k: "default" for k in settings}     if Path(path).exists():        cfg = configparser.ConfigParser()        cfg.read(path)        for key in settings:            if cfg.has_option("app", key):                settings[key] = cfg.get("app", key)                source[key] = "file"     for key in settings:        env = os.environ.get("APP_" + key.upper())        if env is not None:            settings[key] = env            source[key] = "environment"     ap = argparse.ArgumentParser()    for key in settings:        ap.add_argument("--" + key)    args = ap.parse_args(argv)    for key in settings:        value = getattr(args, key)        if value is not None:            settings[key] = value            source[key] = "command line"     return settings, source  if __name__ == "__main__":    settings, source = load("layer.ini")    for key in sorted(settings):        print("%-8s %-22s from the %s" % (key, settings[key], source[key]))PYcd ~/auto && python3 layered.pyhost     localhost              from the defaultport     8000                   from the defaulttimeout  30                     from the default

    Expected resultAll three settings from the default.

    Success conditionYou have a baseline before any layer is applied.

  7. Each layer winning in turn

    Add a file, then an environment variable, then command-line arguments - and watch the source column change.

    bash Example session 
    cd ~/auto && printf '[app]\nhost = from-the-file\nport = 9090\n' > layer.ini && cat layer.ini[app]host = from-the-fileport = 9090cd ~/auto && python3 layered.pyhost     from-the-file          from the fileport     9090                   from the filetimeout  30                     from the defaultcd ~/auto && APP_PORT=7070 python3 layered.pyhost     from-the-file          from the fileport     7070                   from the environmenttimeout  30                     from the defaultcd ~/auto && APP_PORT=7070 python3 layered.py --port 6060 --timeout 5host     from-the-file          from the fileport     6060                   from the command linetimeout  5                      from the command line

    Expected resultThe file overrides two defaults; APP_PORT overrides the file's port; then --port overrides the environment and --timeout overrides the default.

    Success conditionYou can predict which setting wins.

  8. A credential is not configuration

    The one setting that must not be in the file you commit. Read it from the environment, fail clearly if it is missing, and never log the whole thing.

    bash Example session 
    cat > ~/auto/secrets.py <<'PY'import osimport sys token = os.environ.get("API_TOKEN")if not token:    sys.exit("API_TOKEN is not set. Export it, or put it in a file mode 600 outside the repo.") print("using a token of", len(token), "characters, ending", token[-2:])print("and never logging the whole thing")PYpython3 ~/auto/secrets.py; echo "  exit $?"API_TOKEN is not set. Export it, or put it in a file mode 600 outside the repo.  exit 1API_TOKEN=abcdef123456 python3 ~/auto/secrets.py; echo "  exit $?"using a token of 12 characters, ending 56and never logging the whole thing  exit 0

    Expected resultA one-line message and exit 1; then the length and last two characters, and exit 0.

    Success conditionYour script fails usefully when a credential is absent.

  9. A secret in a file, and the permission that matters

    When the environment is not practical - a cron job, a systemd unit - a file outside the repository, readable only by its owner. chmod 600 is not optional.

    bash Example session 
    cd ~/auto && printf 'API_TOKEN=from-a-file-99\n' > .env && chmod 600 .env && ls -l .env-rw------- 1 sysadmin sysadmin 25 Aug 24 03:59 .envcat > ~/auto/readenvfile.py <<'PY'import osfrom pathlib import Path  def load_env_file(path):    """Minimal KEY=value reader. python-dotenv does this properly."""    found = {}    p = Path(path)    if not p.exists():        return found    for line in p.read_text().splitlines():        line = line.strip()        if not line or line.startswith("#") or "=" not in line:            continue        key, _, value = line.partition("=")        found[key.strip()] = value.strip()    return found  values = load_env_file(".env")os.environ.update({k: v for k, v in values.items() if k not in os.environ}) print("loaded from the file:", list(values))print("API_TOKEN now        :", os.environ["API_TOKEN"])PYcd ~/auto && python3 readenvfile.pyloaded from the file: ['API_TOKEN']API_TOKEN now        : from-a-file-99cd ~/auto && API_TOKEN=already-set python3 readenvfile.pyloaded from the file: ['API_TOKEN']API_TOKEN now        : already-set

    Expected result-rw------- on the file, the token loaded from it - and then the environment winning when it is already set.

    Success conditionYou can keep a credential out of the code and out of the repository.

  10. What domain 1 asks about this

    "What type does os.environ.get('PORT') return?" - a string, or None.

    "Is bool(os.environ['DEBUG']) True when DEBUG=false?" - yes, and that is the bug.

    "What does os.environ['MISSING'] raise?" - KeyError.

    "What does cfg.read() do when the file is absent?" - returns an empty list; it does not raise.

    "What is [DEFAULT] for?" - values every section inherits.

    "Which layer should win?" - command line over environment over file over default.

    "Where does a token go?" - the environment, or a mode-600 file outside the repository. Never the config file, never the code.

    That is the shape to copy: one function that returns the resolved settings, so the precedence lives in one place and the rest of the script just reads values.

    guide 8 is next.

    bash Example session 
    rm -f ~/auto/env.py ~/auto/envstrict.py ~/auto/envtypes.py ~/auto/app.ini ~/auto/readini.py ~/auto/layered.py ~/auto/layer.ini ~/auto/secrets.py ~/auto/readenvfile.py ~/auto/.env && ls -A ~/autolabapi.loglabapi.pidlabapi.py

    Expected resultThe scratch directory back to the lab's API fixture.

    Success conditionYour scripts take their settings from outside themselves.

Troubleshooting

Official sources

← All Python Automation guides