CertGrid CertGrid
Troubleshooting·Podman

Podman Error Diagnosis

Every failure on this path named its own cause: `Invalid argument` versus `Permission denied`, pasta versus nginx on port 80, a netns error that still exits 0. A symptom-to-command index, with the wording that tells them apart.

Security and Operations Guide 46 of 47 Advanced

Written against the versions above. Podman follows the distribution here rather than a vendor repository, so the version you get is the one Ubuntu shipped. The commands are stable across 5.x.

Every command on this page ran on podman01.
Server NameIP AddressOSRolesCPURAMHDD
PODMAN01192.168.0.24Ubuntu 26.04 LTSPrimary Container Host2 Core4 GB50 GB

Before you start

  1. The four questions, in order

    When a container is not doing what you expect, these four commands answer almost everything, and the order matters because each narrows the next.

    1. Does it exist and why did it stop? podman ps -a then podman inspect --format '{{.State.ExitCode}}'. Exited (0) finished; Exited (137) was SIGKILLed - usually a stop timeout (guide 15) or the OOM killer.
    2. What did it say? podman logs, and journalctl --user CONTAINER_NAME= when the container is already gone (guide 83).
    3. Is the process there? podman top. A container can be Up with its workers dead.
    4. What did it change? podman diff (guide 13) - and it is how you find the writable paths a hardened container needs.

    Then one more that people reach for last and should reach for early: podman events --since 10m. One timeline of creates, starts, deaths and pulls, and it remembers containers that no longer exist.

    bash Example session 
    podman ps -a --format 'table {{.Names}} {{.Status}}'NAMES       STATUSpodman logs --tail 4 site2026/08/22 11:35:41 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 524288:5242882026/08/22 11:35:41 [notice] 1#1: start worker processes2026/08/22 11:35:41 [notice] 1#1: start worker process 172026/08/22 11:35:41 [notice] 1#1: start worker process 18podman top site pid user commPID         USER        COMMAND1           root        nginx17          nginx       nginx18          nginx       nginxpodman diff siteC /etcC /etc/nginx/conf.dC /etc/nginx/conf.d/default.confA /run/nginx.pidC /var/cache/nginxC /varC /var/cacheA /var/cache/nginx/client_tempA /var/cache/nginx/fastcgi_tempA /var/cache/nginx/proxy_tempA /var/cache/nginx/scgi_tempA /var/cache/nginx/uwsgi_temp

    Expected resultThe four commands answering four different questions about one container.

    Success conditionYou have an order to work through instead of guessing.

  2. Two words that mean different things

    The single most useful distinction on this path, and it comes up everywhere:

    Permission denied - you are not allowed to do this. There may be a capability to add or a threshold to lower. Fixable.

    Invalid argument - the thing you named does not exist in this context. Nothing to grant. Structural.

    Seen on this path:

    | message | guide | meaning | |---|---|---| | chown: Invalid argument | guide 34 | UID past your subuid range - widen it or use a lower UID | | Listen failed for HOST TCP port */80: Permission denied | guide 51 | pasta cannot bind low - sysctl or proxy | | bind() to 0.0.0.0:80 failed (13) | guide 81 | nginx inside, not pasta - add NET_BIND_SERVICE | | chown ... Operation not permitted | guide 81 | missing cap_chown | | sysctl: Read-only file system | guide 80 | /proc/sys is mounted ro - no capability helps | | rm: Permission denied on a container's files | guide 61 | podman unshare rm -rf |

    Note rows two and three: the same port number, two different boundaries. One is pasta on the host, one is nginx in its namespace. Reading which process complained is what tells them apart, and it is why the error text matters more than the error's existence.

    bash Example session 
    podman run --rm docker.io/library/alpine sysctl -w kernel.hostname=nopesysctl: error setting key 'kernel.hostname': Read-only file system[exit 1]podman run --rm --cap-drop=ALL docker.io/library/alpine sh -c 'chown 1:1 /tmp'chown: /tmp: Operation not permitted[exit 1]

    Expected resultA Read-only file system and an Operation not permitted, from two different mechanisms.

    Success conditionYou can tell a missing privilege from a missing mapping by reading the message.

  3. Failures that report success

    The worst class, because nothing looks wrong.

    podman kube down prints Error: and exits 0 (guide 23). A human reading the terminal sees a failure; a script checking $? sees success. Both are right.

    docker compose down exits 1 and leaves a container behind (guide 70). Same underlying netns error, worse outcome.

    replicas: 3 warns and gives you one pod (guide 24). The manifest applied. It did not do what it says.

    A hardened container runs and serves nothing (guide 81). podman ps says Up, podman top shows workers, and every forwarded connection resets.

    podman system connection add succeeds without testing (guide 72). The error waits for first use.

    A rootless service that only starts when you log in (guide 41). It is running every time you look, because looking is what starts it.

    The habit that catches all six: verify from outside, and check the exit code rather than the output. curl from another machine, echo $?, podman ps -a after a teardown. Every one of these was found by a verification step that someone could have skipped.

    bash Example session 
    podman kube down shop.yamlPods stopped:Error: stopping container a5cb6d4e001ba8035d2f3eee8e8fa777d00135c9589b7dc5e59b1ac621fe4574: removing container a5cb6d4e001ba8035d2f3eee8e8fa777d00135c9589b7dc5e59b1ac621fe4574 network: 1 error occurred:	* rootless netns: kill network process: permission denied  Pods removed:b3063debdee5129b9960376fa8cc4c0aacfcd7cc45fb131bffbdf05eb32159b3Secrets removed:Volumes removed:echo "exit code was $?"exit code was 0curl -s -o /dev/null -w 'from host: http=%{http_code}\n' http://localhost:8133from host: http=000[exit 56]

    Expected resultAn Error: line followed by exit code 0, and a hardened container returning no HTTP code.

    Success conditionYou know to distrust a success that you have not checked from outside.

  4. When the answer is the environment

    Some failures are not about your container at all, and recognising them saves the most time.

    State from another engine. A docker0 bridge or DOCKER-USER firewall chains in your networking output mean Docker was here. Purging the packages does not remove the bridge or the netfilter rules until a reboot (guide 1).

    Rootless versus rootful. sudo podman is not a more powerful Podman, it is a different, empty one (guide 14). If adding sudo made an error go away, check whether it also made your images go away.

    The store's own state. After changing /etc/subuid, nothing takes effect until podman system migrate. podman unshare cat /proc/self/uid_map is the check (guide 3).

    The shell. A login shell that is not bash changes error wording and glob handling; captures on this path use bash for exactly that reason.

    And the method that produced most of this path: change one thing at a time and test both sides. The --read-only finding in guide 81 came from four runs differing by one flag, each checked from the host and from inside. That is slower than reasoning about it and it is the only approach that produced an answer.

    bash Example session 
    sudo -n podman info --format 'graphroot={{.Store.GraphRoot}} rootless={{.Host.Security.Rootless}}'graphroot=/var/lib/containers/storage rootless=falsesudo -n podman imagesREPOSITORY  TAG         IMAGE ID    CREATED     SIZEpodman imagesREPOSITORY                     TAG         IMAGE ID      CREATED       SIZEdocker.io/library/nginx        alpine      7bc5ba2f958a  2 days ago    64.2 MBdocker.io/library/alpine       latest      d529dd0c6e55  2 months ago  8.71 MBdocker.io/library/hello-world  latest      e2ac70e7319a  5 months ago  25.5 kB

    Expected resultRoot's empty image list beside your populated one.

    Success conditionYou can recognise a problem that belongs to the environment rather than the container.

Troubleshooting

Official sources

← All Podman guides