Python Encodings, Temp Files and Atomic Writes
Two problems that only show up in production. The first is encoding: read_text() uses whatever the locale says, which is not necessarily what wrote the file. The second is worse - a script that fails half way through a write leaves a truncated file where the good one used to be. This guide shows both, and the function that makes the second impossible.
Files and Data Formats Guide 15 of 39 Intermediate
- Control nodeUbuntu 26.04 LTS
- Python3.14.4
- requests2.34.2
- paramiko5.0.0
- TimeAbout 18 min
PEP 540 means the C locale now enables UTF-8 mode rather than degrading to ASCII, which this guide measures - so the classic locale failure is largely gone. unlink(missing_ok=True) needs 3.8, and os.replace has been the portable rename since 3.3.
- pytest9.1.1
- PyYAML6.0.3
- boto3 / botocore1.43.78
| Server Name | IP Address | OS | Roles | CPU | RAM | HDD |
|---|---|---|---|---|---|---|
| RUNNER01 | 192.168.0.27 | Ubuntu 26.04 LTS | Control node - every script in this path runs here | 2 Core | 4 GB | 50 GB |
This guide includes
Use this for two failures that only appear in production. This matters because a script interrupted mid-write leaves a half-written file behind - and that file is usually the state everything else reads.
- finding which encoding a text file is actually read with, and what the same bytes decode to
- using
errors=for data you do not control - using temporary files and directories that clean themselves up
- seeing the write that leaves a half-file, and replacing it with an atomic one
- learning why it is
os.replaceand notos.rename
Before you start
-
Which encoding a text file is read with
-
And what the same bytes decode to
-
errors=, for data you do not control
-
A temporary file that cleans itself up
-
The write that leaves a half-file behind
-
The atomic version
-
And on a run that succeeds
-
Why os.replace and not os.rename