CertGrid CertGrid
Hands-on Lab·Podman

Podman Capability Dropping and Restoration

Each failure names the next capability: chown, then bind. Four flags later nginx runs on CapEff 4c1 instead of 800405fb. But adding `--read-only` breaks the published port while the container still serves on its own loopback.

Security and Operations Guide 42 of 47 Advanced

Written against the versions above. Podman follows the distribution here rather than a vendor repository, so the version you get is the one Ubuntu shipped. The commands are stable across 5.x.

Every command on this page ran on podman01.
Server NameIP AddressOSRolesCPURAMHDD
PODMAN01192.168.0.24Ubuntu 26.04 LTSPrimary Container Host2 Core4 GB50 GB

Before you start

  1. Find out what it writes before you take anything away

    podman diff on a normally-running nginx lists every path it created:

    A /run/nginx.pid
A /var/cache/nginx/client_temp
A /var/cache/nginx/fastcgi_temp
...

    Two directories: /run and /var/cache/nginx. That is the complete list of what needs to be writable, obtained by measurement rather than by reading the image's documentation - which frequently does not say.

    Do this first for any image you intend to harden. Guessing produces a container that starts and fails an hour later on a path nobody thought about.

    bash Example session 
    podman run -d --name plain docker.io/library/nginx:alpine330cc78e115254ad6fc5a2b7ada2a742a6167ca6d30f1e57b42d9947ec54cb22podman diff plainC /etcC /etc/nginx/conf.dC /etc/nginx/conf.d/default.confA /run/nginx.pidC /var/cache/nginxC /varC /var/cacheA /var/cache/nginx/client_tempA /var/cache/nginx/fastcgi_tempA /var/cache/nginx/proxy_tempA /var/cache/nginx/scgi_tempA /var/cache/nginx/uwsgi_temppodman rm -f plainplain

    Expected resultA list of added paths under /run and /var/cache/nginx.

    Success conditionYou have the image's writable paths as a measured list.

  2. Drop everything and read the failure

    --cap-drop=ALL, with tmpfs at both paths from step 1. It exits 1:

    [emerg] 1#1: chown("/var/cache/nginx/client_temp", 101) failed
  (1: Operation not permitted)

    The error names the capability. nginx wants to give its cache directories to uid 101 so the workers can use them, and chown needs cap_chown, which you just dropped.

    This is the methodology and it is worth doing in this order. Starting from nothing and adding back what the failures demand gives you a minimal, justified set where every capability is there for a reason you have read. Starting from the default eleven and guessing which to remove gives you a set nobody can explain.

    bash Example session 
    podman run -d --name h1 --read-only --tmpfs /var/cache/nginx --tmpfs /run --cap-drop=ALL -p 8110:80 docker.io/library/nginx:alpineb92b3ee66f2faa89b1124ac787ddc633c65432846945de99d38b284e246eabd1podman inspect h1 --format 'status={{.State.Status}} exit={{.State.ExitCode}}'status=exited exit=1podman logs h1 2>&1 | tail -22026/08/22 12:50:42 [emerg] 1#1: chown("/var/cache/nginx/client_temp", 101) failed (1: Operation not permitted)nginx: [emerg] chown("/var/cache/nginx/client_temp", 101) failed (1: Operation not permitted)podman rm -f h1h1

    Expected resultstatus=exited exit=1 and a chown ... Operation not permitted emerg line.

    Success conditionThe container failed with a message naming exactly what it needs.

  3. Add back one, and read the next failure

    --cap-add=CHOWN. It gets further and stops again:

    [emerg] 1#1: bind() to 0.0.0.0:80 failed (13: Permission denied)

    A different failure, naming a different need. Port 80 is privileged inside the container too, and binding it needs cap_net_bind_service - which is in Podman's default eleven and which --cap-drop=ALL removed.

    Note this is a separate thing from guide 51. That was pasta failing to bind 80 on the host; this is nginx failing to bind 80 in its own namespace. Same number, two different boundaries, and it is worth keeping them apart.

    Add the remaining three - NET_BIND_SERVICE for the bind, SETUID and SETGID for dropping to the worker user - and it runs:

    PID  USER   COMMAND
1    root   nginx
16   nginx  nginx
17   nginx  nginx

CapEff: 00000000000004c1

    4c1 against the default 800405fb. Four capabilities instead of eleven, each one traceable to a failure you read. touch /nope still gives Read-only file system, so the root filesystem is immutable and the workers run unprivileged.

    bash Example session 
    podman run -d --name h2 --read-only --tmpfs /var/cache/nginx --tmpfs /run --cap-drop=ALL --cap-add=CHOWN -p 8111:80 docker.io/library/nginx:alpineace3e17cf3711a8bbd9311b6574ba6df816a0dc21c815a609512aa0a2f085783podman inspect h2 --format 'status={{.State.Status}} exit={{.State.ExitCode}}'status=exited exit=1podman logs h2 2>&1 | tail -22026/08/22 12:50:43 [emerg] 1#1: bind() to 0.0.0.0:80 failed (13: Permission denied)nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)podman rm -f h2h2podman top h3 pid user commPID         USER        COMMAND1           root        nginx16          nginx       nginx17          nginx       nginxpodman exec h3 grep -E '^CapEff' /proc/self/statusCapEff:	00000000000004c1podman exec h3 sh -c 'touch /nope'touch: /nope: Read-only file system[exit 1]podman inspect h3 --format 'ro={{.HostConfig.ReadonlyRootfs}} dropped={{len .HostConfig.CapDrop}}'ro=true dropped=7

    Expected resultA bind failure with CHOWN alone, then a running container on CapEff 4c1 with a read-only root.

    Success conditionYou have a container whose every capability you can justify.

  4. And the flag that quietly breaks the port

    The hardened container above is running, its workers are up, and podman top looks perfect. It does not serve traffic:

    $ curl http://localhost:8133
from host: http=000        (exit 56, connection reset)

$ podman exec r3 wget -qO- http://localhost/
<!DOCTYPE html>            (works)

    Serving fine on its own loopback, resetting every forwarded connection. So the question is which flag did it - and the answer comes from changing one at a time and testing both sides:

    | flags | from host | inside | |---|---|---| | none | 200 | serves | | --cap-drop=ALL plus the four adds | 200 | serves | | --tmpfs /run | 200 | - | | --read-only --tmpfs ... | 000 | serves |

    It is --read-only. Not the capabilities - those serve 200. Not --tmpfs /run - that serves 200 on its own. A read-only root filesystem and a published port do not work together on rootless Podman 5.7.0 with pasta, and the container gives no sign of it: it runs, it logs normally, and it answers itself.

    What to do about it. Check from outside. A container that starts is not a container that serves, and this whole finding exists because a curl was in the capture. If you harden something, request it from another machine before believing it.

    Then choose: keep the capability drops and the tmpfs mounts, which cost nothing here and gave the 200s; and get immutability another way - a rootful container where this combination does work, --network host so nothing is forwarded, or accepting a writable root and relying on the capability set. --read-only is the one piece of standard hardening advice that does not transfer to rootless unchanged.

    bash Example session 
    podman run -d --name r0 -p 8130:80 docker.io/library/nginx:alpine503984d6c6d918c3b18b01885a57a5c509002f4531d808554a06fbf91845ccb8curl -s -o /dev/null -w 'from host: http=%{http_code}\n' http://localhost:8130from host: http=200podman run -d --name r1 --cap-drop=ALL --cap-add=CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=NET_BIND_SERVICE -p 8131:80 docker.io/library/nginx:alpine0862b0b0d6b6d86d28e3e9df2d1fba62f29a0d0fcb148441cd2b1a840e0cb976curl -s -o /dev/null -w 'from host: http=%{http_code}\n' http://localhost:8131from host: http=200podman run -d --name r2 --tmpfs /run -p 8132:80 docker.io/library/nginx:alpinedaabd4ea7511016eedc680501114c95ffc66e76508b48822bafb06abbfc3f636curl -s -o /dev/null -w 'from host: http=%{http_code}\n' http://localhost:8132from host: http=200podman run -d --name r3 --read-only --tmpfs /var/cache/nginx --tmpfs /run -p 8133:80 docker.io/library/nginx:alpinebe260617113a7e040816425733174ac39b79885a81db15dc9d8798e55d8ae850curl -s -o /dev/null -w 'from host: http=%{http_code}\n' http://localhost:8133from host: http=000[exit 56]podman exec r3 wget -qO- -T2 http://localhost/ 2>/dev/null | head -1<!DOCTYPE html>podman rm -f r3r3

    Expected result200 for the first three configurations and 000 for the read-only one, which still serves its own loopback.

    Success conditionYou changed one flag at a time and can name the one that broke it.

Troubleshooting

Official sources

← All Podman guides