CertGrid CertGrid
Troubleshooting·Docker

Docker Production Troubleshooting

The failures you will actually meet, each reproduced deliberately so you recognise the exact wording. Exit codes 125 through 137, restart loops, permission denials and the port conflict that reads like a networking bug.

Operations and Troubleshooting Guide 33 of 46 Advanced

Tested on the versions above. Error wording changes between Docker versions. The exit codes and the diagnostic method are stable.

One Docker host is all this guide needs. Nothing here depends on a second machine, and the hardware above is modest on purpose - a 2 core, 4 GB VM runs everything in this path.
Server NameIP AddressOSRolesCPURAMHDD
DOCKER01192.168.0.21Ubuntu 26.04 LTSDocker Host2 Core4 GB50 GB

Before you start

  1. Read the exit code before anything else

    Docker's exit codes are a diagnosis, not noise. 125 means the DAEMON refused - the container never started. 126 means the command was found but could not be executed. 127 means it was not found at all. Anything else is your application's own exit status, passed through unchanged. Learning these four cases removes most of the guesswork.

    bash 
    docker run --rm alpine:3.22 sh -c "exit 42"; echo host-saw=$?host-saw=42# your exit status reaches the host untouched - 42 is not a Docker code

    Expected resultThe application's own status, unmodified.

    Success conditionYou can separate "Docker refused" from "the program failed". Everything below is one of those two.

  2. 127 and 126: the command is wrong

    These two look alike and mean different things. 127 is a missing executable - a typo, or a binary that is not in a slim image. 126 is a file that exists but is not executable, which is usually a script missing its executable bit or its interpreter line. Note both errors mention the OCI runtime, which sends people looking in the wrong place.

    bash 
    docker run --rm alpine:3.22 notacommandunable to start container process: error during container init: exec: "notacommand": executable file not found in $PATH# exit 127docker run --rm alpine:3.22 /etc/hostnameunable to start container process: error during container init: exec: "/etc/hostname": permission denied# exit 126

    Expected resultTwo different messages, 127 then 126.

    Success conditionYou can tell them apart from the wording alone. For 127 in a slim image, check the binary actually exists: docker run --rm IMAGE which yourcommand.

  3. 125: the daemon refused before your code ran

    125 always means the container was never created. The two you will meet most are a missing image and a port already in use. The port message is long and mentions networking drivers and endpoints, which reads like a network fault - the operative words are at the very end.

    bash 
    docker run --rm nosuchimage:doesnotexistpull access denied for nosuchimage, repository does not exist or may require 'docker login'docker run -d --name cg-p2 -p 8099:80 nginx:alpinedriver failed programming external connectivity on endpoint cg-p2: Bind for 0.0.0.0:8099 failed: port is already allocated

    Expected resultBoth refused with 125, neither container created.

    Verify it worked

    bash 
    docker ps --filter publish=8099 --format "{{.Names}} {{.Ports}}"# names the container already holding the port

    Success conditionYou read to the end of the message. "Port is already allocated" is a conflict, not a driver problem - and note a private-registry image can produce the same "pull access denied" wording when you are simply not logged in.

  4. 137: killed, and how to tell why

    137 is 128 plus signal 9. It appears for an out-of-memory kill, for docker kill, and for a container that ignored SIGTERM during shutdown and got killed after the grace period. Only the OOMKilled flag distinguishes the first from the others.

    bash 
    docker inspect cg-oom --format "oomkilled={{.State.OOMKilled}} exit={{.State.ExitCode}} status={{.State.Status}}"oomkilled=true exit=137 status=exited

    Expected resultThe flag that turns an ambiguous code into a diagnosis.

    Success conditionYou check the flag rather than assuming. If it is false, look at your stop grace period and whether the process handles SIGTERM.

  5. The restart loop, and how to stop it

    A container with an unbounded restart policy that fails immediately will spin forever. docker ps shows it as Restarting rather than Up, and the restart count climbs every few seconds. The trap is that docker logs shows only the current attempt, and docker rm -f can race the restart. docker update --restart=no breaks the cycle first, then you can work calmly.

    bash Example session 
    docker inspect cg-loop --format "restarts={{.RestartCount}} status={{.State.Status}}"restarts=7 status=restartingdocker ps --filter name=cg-loop --format "{{.Names}} {{.Status}}"cg-loop Restarting (1) Less than a second ago

    Expected resultA climbing restart count and the Restarting status, then a container that stays put.

    Verify it worked

    bash 
    # stop the loop BEFORE investigating, so the container holds stilldocker update --restart=no cg-loopcg-loop

    Success conditionThe loop is stopped without destroying the evidence. Restarting (1) also tells you the exit code it keeps failing with.

  6. Permission denied on a volume

    The most common non-obvious failure in production. A new named volume is created root-owned, so a container running as a non-root user cannot write to it - and the image works fine in development where it runs as root. The error is a bare permission denied with no mention of ownership.

    bash 
    docker run --rm --user 1000:1000 -v cg-perm:/data alpine:3.22 sh -c "touch /data/f && echo wrote ok"touch: /data/f: Permission denieddocker run --rm -v cg-perm:/data alpine:3.22 ls -ld /datadrwxr-xr-x    2 root     root          4096 Aug 20 08:20 /data

    Expected resultThe write refused, and the directory owned by root.

    Success conditionYou diagnosed it by listing ownership rather than guessing. Fix by chowning the mount point in an init step, or by running the service as the uid that owns the data.

  7. exec on a container that is not running

    docker exec requires a running container, and the error says so plainly - but it is easy to misread as the container being missing. A stopped container still exists; you simply cannot execute in it. To inspect a container that will not stay up, override its entrypoint and start a shell instead.

    bash 
    docker stop cg-p1 >/dev/null && docker exec cg-p1 whoamiError response from daemon: container ea07324cba54... is not running# to debug an image that will not start, bypass its entrypoint:docker run --rm -it --entrypoint sh IMAGE

    Expected resultA clear "is not running", not "no such container".

    Success conditionYou distinguish stopped from absent. docker ps -a shows the former; only the latter is really gone.

  8. A method that works when nothing above matches

    When the failure is unfamiliar, go in this order. It moves from cheapest to most invasive and each step narrows the next.

    bash 
    docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"# 1. state and exit code - is it Up, Exited, or Restartingdocker logs --tail 50 NAME# 2. what the application said before it stoppeddocker inspect NAME --format "{{.State.Status}} {{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}"# 3. the daemon's view, including errors that never reach the logsdocker events --since 10m --filter container=NAME# 4. the sequence of what happened, when the above is not enough

    Expected resultFour commands you can run from memory.

    Success conditionYou have an order to work in. Most failures are answered by the first two; State.Error catches several that produce no logs at all.

Troubleshooting

Official sources

← All Docker guides