CertGrid CertGrid
Troubleshooting·Docker

Docker Daemon and Container Events

The event stream is the timeline logs do not give you. Reproduce a health failure, an external kill and two memory kills, then use events to tell apart three containers that all exit 137.

Operations and Troubleshooting Guide 34 of 46 Advanced

Tested on the versions above. Every line below comes from one capture run on a real host: the containers were created together, one was killed deliberately, and every query covers that same window. Timestamps and container IDs are from that run.

One Docker host is all this guide needs. Nothing here depends on a second machine, and the hardware above is modest on purpose - a 2 core, 4 GB VM runs everything in this path.
Server NameIP AddressOSRolesCPURAMHDD
DOCKER01192.168.0.21Ubuntu 26.04 LTSDocker Host2 Core4 GB50 GB

Before you start

  1. Events are a timeline, logs are a transcript

    Container logs tell you what an application said. Events tell you what the daemon did, in order, with timestamps. When something restarted at 03:14 and you need to know whether a health check failed first, or whether the kernel killed it, only events answer that. Bound the window with --since and --until; with neither, the command streams live and blocks.

    bash 
    docker events --since 30m --until "$END" --filter type=container --format "{{.Time}} {{.Actor.Attributes.name}} {{.Action}}" | head -41787232513 cg-ev-health create1787232513 cg-ev-health start1787232513 cg-ev-kill create1787232513 cg-ev-kill start# --since accepts 30m, 1h, or an absolute timestamp# with no --until the command streams live instead of returning

    Expected resultA timeline of container actions, oldest first.

    Success conditionYou can query history rather than only watching live. Historical queries are what you need during an incident; live streaming is for while you reproduce something.

  2. Reproduce three different failures

    Reading events is only worth practising against events worth reading. These three cover what actually happens in production: a container failing its health check, one killed from outside, and two hitting a memory limit in different ways. Note the timestamps are recorded first so every query below covers exactly this window.

    bash 
    START=$(date -u +%Y-%m-%dT%H:%M:%S)docker run -d --name cg-ev-health --health-cmd "exit 1" --health-interval 2s --health-retries 2 nginx:alpinedocker run -d --name cg-ev-kill alpine:3.22 sleep 300docker run -d --name cg-ev-oom --memory 64m --memory-swap 64m alpine:3.22 sh -c 'tail /dev/zero'docker run -d --name cg-ev-shm --memory 64m --shm-size 256m alpine:3.22 sh -c 'head -c 200000000 /dev/zero > /dev/shm/fill'sleep 14; docker kill cg-ev-killcg-ev-killEND=$(date -u +%Y-%m-%dT%H:%M:%S)docker ps -a --filter name=cg-ev- --format "{{.Names}} {{.Status}}"cg-ev-shm Exited (137) 16 seconds agocg-ev-oom Exited (137) 17 seconds agocg-ev-kill Exited (137) 3 seconds agocg-ev-health Up 17 seconds (unhealthy)

    Expected resultThree containers exited and one running but unhealthy.

    Success conditionThree of the four exited 137. That is the problem this guide solves - the same exit code for three different causes. tail /dev/zero reads endlessly into memory; the other writes past the memory limit into a tmpfs.

  3. Read the whole timeline first

    Before filtering, look at the sequence. Every lifecycle transition is here in order, and two things stand out: the memory kills produce an explicit oom action before their die, and a health check is noisy - exec_create, exec_start and exec_die on every single interval.

    bash 
    docker events --since "$START" --until "$END" --filter type=container --format "{{.Time}} {{.Actor.Attributes.name}} {{.Action}}" | head -161787232513 cg-ev-health create1787232513 cg-ev-health start1787232513 cg-ev-kill create1787232513 cg-ev-kill start1787232513 cg-ev-oom create1787232513 cg-ev-oom start1787232513 cg-ev-oom oom1787232513 cg-ev-shm create1787232514 cg-ev-oom die1787232514 cg-ev-shm start1787232514 cg-ev-shm oom1787232514 cg-ev-shm die1787232515 cg-ev-health exec_create: /bin/sh -c exit 11787232515 cg-ev-health exec_start: /bin/sh -c exit 11787232515 cg-ev-health exec_die1787232517 cg-ev-health exec_create: /bin/sh -c exit 1

    Expected resultSixteen lines covering creation, the two oom events, the deaths and the health-check execs.

    Success conditionYou can see cause before effect. cg-ev-oom oom lands immediately before cg-ev-oom die - that ordering is the evidence, not an inference.

  4. Every container that stopped, and why

    This one query answers most "what happened overnight" questions. The exit code travels as an attribute of the die event, so you get the outcome without inspecting anything.

    bash 
    docker events --since "$START" --until "$END" --filter event=die --format "{{.Actor.Attributes.name}} died exitCode={{index .Actor.Attributes \"exitCode\"}}"cg-ev-oom died exitCode=137cg-ev-shm died exitCode=137cg-ev-kill died exitCode=137

    Expected resultThree containers, all exit 137.

    Success conditionThis is the trap. Exit 137 means the process received SIGKILL. It says nothing about who sent it - the kernel's OOM killer and an operator with docker kill produce the identical code.

  5. Tell an OOM apart from a kill

    Two more filters separate them completely. The oom event fires only when the kernel's out-of-memory killer acts on the container's cgroup. The kill event fires when something asked the daemon to stop it. Each container's own stream shows exactly one of the two.

    bash 
    docker events --since "$START" --until "$END" --filter event=oom --format "oom: {{.Actor.Attributes.name}}"oom: cg-ev-oomoom: cg-ev-shm# cg-ev-kill is absent - it was not a memory problem

    Expected resultTwo of the three named, and the third missing.

    Verify it worked

    bash Example session 
    docker events --since "$START" --until "$END" --filter container=cg-ev-oom --format "{{.Action}}"createstartoomdie docker events --since "$START" --until "$END" --filter container=cg-ev-kill --format "{{.Action}}"createstartkilldie

    Success conditionYou have attribution rather than a guess. Look at the individual streams and the difference is explicit.

  6. The container's own state agrees

    Events are the timeline; the container object carries the same conclusion as a flag. Use whichever is available - events after a container has been removed, inspect while it still exists.

    bash 
    docker inspect cg-ev-oom --format "oomkilled={{.State.OOMKilled}} exit={{.State.ExitCode}}"oomkilled=true exit=137docker inspect cg-ev-shm --format "oomkilled={{.State.OOMKilled}} exit={{.State.ExitCode}}"oomkilled=true exit=137docker inspect cg-ev-kill --format "oomkilled={{.State.OOMKilled}} exit={{.State.ExitCode}}"oomkilled=false exit=137# same exit code on all three, and OOMKilled is what separates them

    Expected resultTwo containers with oomkilled=true and one with false, all exit 137.

    Success conditionBoth memory kills set the flag regardless of how the memory was consumed - one filled anonymous memory, the other a tmpfs, and the cgroup limit caught both. See guide 30 for setting those limits deliberately.

  7. Catch degradation that never restarts anything

    A service that goes unhealthy and recovers repeatedly never restarts, so nothing watching container state will notice. health_status is the only place it surfaces, and it is worth alerting on.

    bash 
    docker events --since "$START" --until "$END" --filter event=health_status --format "{{.Actor.Attributes.name}} -> {{.Action}}" | head -4cg-ev-health -> health_status: unhealthy# flapping appears here as repeated healthy / unhealthy transitions

    Expected resultThe transition to unhealthy as its own event.

    Success conditionYou have a signal for a container that is failing without dying. Pair it with guide 26.

  8. One query for an incident

    During an incident you rarely want one event type. Filters of the same kind are OR-ed, so this single command gives deaths, memory kills and health transitions on one timeline - which is usually the whole picture.

    bash 
    docker events --since "$START" --until "$END" --filter event=die --filter event=oom --filter event=health_status --format "{{.Action}} {{.Actor.Attributes.name}}"oom cg-ev-oomdie cg-ev-oomoom cg-ev-shmdie cg-ev-shmhealth_status: unhealthy cg-ev-healthdie cg-ev-kill

    Expected resultAll three failure classes interleaved in the order they happened.

    Success conditionThe oom lines sit immediately before their die lines, and the health transition sits on its own. This is the query to reach for first.

  9. Events cover more than containers

    Networks, volumes, images, plugins and the daemon itself all emit events. Network attach and detach are the ones you will want when a container cannot reach something and you need to know whether it ever joined the network at all.

    bash 
    docker events --since "$START" --until "$END" --filter type=network --format "{{.Type}} {{.Action}} {{.Actor.Attributes.name}}" | head -4network connect bridgenetwork connect bridgenetwork connect bridgenetwork disconnect bridge# also available: --filter type=volume, type=image, type=daemon

    Expected resultAttach and detach events for the bridge network.

    Success conditionYou know the stream is broader than containers. With no type filter it shows everything, which is unreadable live but useful scoped to a window.

  10. Clean up

    Remove the four containers created to generate these events.

    bash 
    docker rm -f cg-ev-health cg-ev-oom cg-ev-kill cg-ev-shmcg-ev-healthcg-ev-oomcg-ev-killcg-ev-shmdocker ps -aq | wc -l0

    Expected resultThe four names echoed back and no containers left.

    Success conditionThe host is back where it started.

Troubleshooting

Official sources

← All Docker guides