CertGrid CertGrid
Hands-on Lab·Python Automation for IT

Python Error Handling for Scheduled Jobs

Code you are watching can fail with a traceback and you will read it. Code that runs at 3am needs to fail in a way that tells somebody what happened, keeps going where it sensibly can, cleans up after itself, and exits with a status the scheduler acts on. This guide builds that, and finishes with a real cron entry - installed and removed inside the capture - showing exactly how a scheduled process differs from your shell.

Scripting Foundations Guide 9 of 39 Intermediate

Written against the versions above. Nothing here is version-sensitive. The cron environment shown in the last step is this machine's - Ubuntu 26.04 with the standard `cron` package - and the guide says which parts vary by distribution.

Everything on this page runs on the control node. Any machine with Python 3 will do.
Server NameIP AddressOSRolesCPURAMHDD
RUNNER01192.168.0.27Ubuntu 26.04 LTSControl node - every script in this path runs here2 Core4 GB50 GB

Before you start

  1. An unhandled exception is honest, and not enough

    A missing configuration file, unhandled. The traceback is accurate: it names the exception, the file and the line.

    What it does not do is tell a person reading a cron mail what to *do*.

    bash Example session 
    cat > ~/auto/bare.py <<'PY'import jsonfrom pathlib import Path config = json.loads(Path("missing.json").read_text())print(config)PYcd ~/auto && python3 bare.py; echo "  exit $?"Traceback (most recent call last):  File "/home/sysadmin/auto/bare.py", line 4, in <module>    config = json.loads(Path("missing.json").read_text())                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^  File "/usr/lib/python3.14/pathlib/__init__.py", line 788, in read_text    with self.open(mode='r', encoding=encoding, errors=errors, newline=newline) as f:         ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/lib/python3.14/pathlib/__init__.py", line 772, in open    return io.open(self, mode, buffering, encoding, errors, newline)           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^FileNotFoundError: [Errno 2] No such file or directory: 'missing.json'  exit 1

    Expected resultA four-frame traceback ending in FileNotFoundError, and exit 1.

    Success conditionYou can see the default behaviour before improving it.

  2. The same failure, said in one line

    Catch the specific exceptions, say what went wrong in the terms the caller cares about, and exit non-zero. Three outcomes below: missing, malformed, and fine.

    bash Example session 
    cat > ~/auto/oneline.py <<'PY'import jsonimport sysfrom pathlib import Path path = Path("missing.json")try:    config = json.loads(path.read_text())except FileNotFoundError:    sys.exit("config not found: %s" % path)except json.JSONDecodeError as exc:    sys.exit("config is not valid JSON: %s: %s" % (path, exc)) print(config)PYcd ~/auto && python3 oneline.py; echo "  exit $?"config not found: missing.json  exit 1cd ~/auto && printf 'not json at all\n' > missing.json && python3 oneline.py; echo "  exit $?"config is not valid JSON: missing.json: Expecting value: line 1 column 1 (char 0)  exit 1cd ~/auto && printf '{"ok": true}\n' > missing.json && python3 oneline.py; echo "  exit $?"{'ok': True}  exit 0

    Expected resultconfig not found: missing.json; then config is not valid JSON with the parser's own detail; then the parsed dictionary. Exit 1, 1, 0.

    Success conditionYour script explains its own failures.

  3. Catch the narrow thing, not everything

    This is the highest-value habit in the domain. The loop below catches Exception so that one unreachable host does not stop the run - and in doing so it swallows a genuine bug.

    One host times out. The other has int("not a number") in its path.

    bash Example session 
    cat > ~/auto/toobroad.py <<'PY'import logging logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(message)s")log = logging.getLogger("job") hosts = ["web01", "web02"] for host in hosts:    try:        if host == "web01":            raise TimeoutError("no answer in 5s")        value = int("not a number")        # a real bug, not a host problem    except Exception as exc:        log.error("%s failed: %s", host, exc) log.info("finished, and reported success")PYcd ~/auto && python3 toobroad.py; echo "  exit $?"ERROR    web01 failed: no answer in 5sERROR    web02 failed: invalid literal for int() with base 10: 'not a number'INFO     finished, and reported success  exit 0

    Expected resultTwo ERROR lines that look alike, finished, and reported success, and exit 0.

    Success conditionYou can see a bug disguised as an expected failure.

  4. The same loop, narrowed and counted

    except TimeoutError: catches the thing that was expected. The bug is left alone, so it crashes with its traceback - and the expected failure is counted and reflected in the exit status.

    bash Example session 
    cat > ~/auto/narrow.py <<'PY'import loggingimport sys logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(message)s")log = logging.getLogger("job") hosts = ["web01", "web02"]failed = 0 for host in hosts:    try:        if host == "web01":            raise TimeoutError("no answer in 5s")        value = int("not a number")    except TimeoutError as exc:        log.error("%s unreachable: %s", host, exc)        failed += 1 log.info("%d ok, %d failed", len(hosts) - failed, failed)sys.exit(1 if failed else 0)PYcd ~/auto && python3 narrow.py; echo "  exit $?"ERROR    web01 unreachable: no answer in 5sTraceback (most recent call last):  File "/home/sysadmin/auto/narrow.py", line 14, in <module>    value = int("not a number")ValueError: invalid literal for int() with base 10: 'not a number'  exit 1

    Expected resultThe timeout logged and counted, then the ValueError crashing with a traceback - and exit 1.

    Success conditionExpected failures are handled and bugs are not hidden.

  5. A top-level handler, so nothing escapes silently

    One try around the work, with the expected failure and the unexpected one handled differently: a message and exit 1 for the first, a full traceback and a distinct status for the second.

    bash Example session 
    cat > ~/auto/toplevel.py <<'PY'import loggingimport sys logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(message)s")log = logging.getLogger("job")  def work(mode):    if mode == "bug":        return int("not a number")    if mode == "expected":        raise TimeoutError("host did not answer")    return "fine"  def main():    mode = sys.argv[1]    try:        log.info("result: %s", work(mode))    except TimeoutError as exc:        log.error("expected failure: %s", exc)        return 1    except Exception:        log.exception("unexpected failure - this is a bug, not a host problem")        return 70    return 0  if __name__ == "__main__":    sys.exit(main())PYcd ~/auto && python3 toplevel.py ok; echo "  exit $?"INFO     result: fine  exit 0cd ~/auto && python3 toplevel.py expected; echo "  exit $?"ERROR    expected failure: host did not answer  exit 1cd ~/auto && python3 toplevel.py bug 2>&1 | tail -8; echo "  (traceback kept)"ERROR    unexpected failure - this is a bug, not a host problemTraceback (most recent call last):  File "/home/sysadmin/auto/toplevel.py", line 19, in main    log.info("result: %s", work(mode))                           ~~~~^^^^^^  File "/home/sysadmin/auto/toplevel.py", line 10, in work    return int("not a number")ValueError: invalid literal for int() with base 10: 'not a number'  (traceback kept)cd ~/auto && python3 toplevel.py bug >/dev/null 2>&1; echo "  exit $?"  exit 70

    Expected resultExit 0 on success; a one-line error and exit 1 for the timeout; a logged traceback and exit 70 for the bug.

    Success conditionA caller can tell an operational failure from a bug by the status alone.

  6. Cleanup that happens whatever else does not

    finally runs on the way out whether the block succeeded, failed, or raised something nobody caught. A lock file is the clearest case: leave one behind and the next run refuses to start.

    bash Example session 
    cat > ~/auto/cleanup2.py <<'PY'import loggingimport sysfrom pathlib import Path logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(message)s")log = logging.getLogger("job") lock = Path("job.lock")lock.write_text(str(sys.argv))log.info("lock created: %s", lock.exists()) try:    if sys.argv[1] == "fail":        raise RuntimeError("something went wrong halfway")    log.info("work finished")finally:    lock.unlink(missing_ok=True)    log.info("lock removed: %s", not lock.exists())PYcd ~/auto && python3 cleanup2.py ok; echo "  exit $?"; ls job.lock 2>&1 | tail -1INFO     lock created: TrueINFO     work finishedINFO     lock removed: True  exit 0ls: cannot access 'job.lock': No such file or directorycd ~/auto && python3 cleanup2.py fail 2>&1 | tail -4; ls job.lock 2>&1 | tail -1Traceback (most recent call last):  File "/home/sysadmin/auto/cleanup2.py", line 14, in <module>    raise RuntimeError("something went wrong halfway")RuntimeError: something went wrong halfwayls: cannot access 'job.lock': No such file or directory

    Expected resultThe lock created and removed on the clean run; on the failing run, lock removed: True appears before the traceback.

    Success conditionYour script does not leave state behind when it dies.

  7. What cron actually gives a script

    Now the reason all of this matters. The same probe script, run twice: once from an interactive shell, and once by a real cron entry installed and removed inside this capture.

    Compare the two outputs line by line.

    bash Example session 
    cat > ~/auto/cronprobe.py <<'PY'import osimport sysfrom pathlib import Path out = Path.home() / "auto" / "cron-out.txt"lines = [    "interpreter : %s" % (sys.executable or "(empty)"),    "cwd         : %s" % os.getcwd(),    "PATH        : %s" % os.environ.get("PATH", "(unset)"),    "HOME        : %s" % os.environ.get("HOME", "(unset)"),    "variables   : %d" % len(os.environ),    "has TERM    : %s" % ("TERM" in os.environ),    "stdout a tty: %s" % sys.stdout.isatty(),]out.write_text("\n".join(lines) + "\n")PYcd ~/auto && python3 cronprobe.py && echo "--- run from an interactive shell ---" && cat cron-out.txt--- run from an interactive shell ---interpreter : /usr/bin/python3cwd         : /home/sysadmin/autoPATH        : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/binHOME        : /home/sysadminvariables   : 17has TERM    : Falsestdout a tty: Falserm -f ~/auto/cron-out.txt && (crontab -l 2>/dev/null; echo "* * * * * /usr/bin/python3 /home/sysadmin/auto/cronprobe.py") | crontab - && crontab -l | tail -1* * * * * /usr/bin/python3 /home/sysadmin/auto/cronprobe.pysleep 70; echo "--- the same script, run by cron ---"; cat ~/auto/cron-out.txt--- the same script, run by cron ---interpreter : /usr/bin/python3cwd         : /home/sysadminPATH        : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/binHOME        : /home/sysadminvariables   : 6has TERM    : Falsestdout a tty: False

    Expected resultInteractively: cwd /home/sysadmin/auto, 17 variables. Under cron: cwd /home/sysadmin, 6 variables. Same interpreter, same PATH.

    Success conditionYou know what is different about a scheduled run.

  8. The shape to copy

    Everything on this page, in one outline:

    #!/home/sysadmin/autoenv/bin/python
import logging, sys
from pathlib import Path

log = logging.getLogger("job")
HERE = Path(__file__).resolve().parent    # not the cwd

def main():
    args = parse_args()
    logging.basicConfig(...)

    failed = 0
    for item in load(HERE / "inventory.json"):
        try:
            do_one(item)
        except (TimeoutError, OSError) as exc:   # expected
            log.error("%s: %s", item, exc)
            failed += 1

    log.info("%d ok, %d failed", total - failed, failed)
    return 1 if failed else 0

if __name__ == "__main__":
    try:
        sys.exit(main())
    except Exception:
        log.exception("unexpected failure")
        sys.exit(70)

    Seven decisions in twenty lines: absolute interpreter, paths anchored to the file, logging not print, narrow clauses in the loop, a count rather than a bare boolean, a binary exit status, and a top-level handler that keeps the traceback and gives bugs their own code.

    That completes domain 1 - 19% of the mock, and the track everything else assumes. guide 10 starts domain 2.

    bash Example session 
    crontab -l | grep -v cronprobe | crontab - ; crontab -l 2>&1 | tail -2; echo "  (cron entry removed)"  (cron entry removed)

    Expected resultThe cron entry removed - this capture cleans up after itself too.

    Success conditionDomain 1 is complete.

Troubleshooting

Official sources

← All Python Automation guides