Troubleshooting OOMKilled and Exit Code 137

Troubleshooting / Guide 91 of 103 / Beginner

• Kubernetes1.36.4
• Runtimecontainerd 2.2.6
• Cluster4 nodes
• CNICalico v3.32.1
• TimeAbout 25 min
• Reviewed21 August 2026

Written against the versions above. The kill is done by the kernel cgroup OOM killer, not by Kubernetes. That is why it is immediate and unappealable.

Before you start

• The requests, limits and QoS guide. This is what happens when a memory limit is reached.
• The CrashLoopBackOff guide, for exit codes and lastState.
• The emptyDir guide, for why writing to /dev/shm counts as memory.

1. 1

   Ask for more memory than the limit allows

   The container writes 200MB to /dev/shm against a 64Mi limit. /dev/shm is a tmpfs, so those bytes are memory: this is a compact way to allocate for real without needing a memory-hungry program.

   The status is OOMKilled directly in kubectl get pods, which makes it the most self-explanatory failure in Kubernetes. No log reading required.

   And note RESTARTS 0 with restartPolicy: Never. Under the default Always this would be a crash loop, cycling OOMKilled and CrashLoopBackOff with a climbing restart count, which is how it usually appears in the wild: a Pod restarting every few minutes, and the exit code the only clue.

   The echo survived at the end of that command never runs. That is characteristic and important: the process is killed with SIGKILL, so there is no cleanup, no flush, no shutdown handler and no final log line. An OOM kill cannot be caught or handled. Anything the application was part-way through is simply abandoned, which for a database mid-write is how corruption happens.

   kubectl apply -f - <<'EOF'apiVersion: v1kind: Podmetadata:  name: hungry  namespace: tshspec:  restartPolicy: Never  containers:  - name: app    image: busybox:1.36    command: ["sh", "-c", "echo allocating; dd if=/dev/zero of=/dev/shm/fill bs=1M count=200; echo survived"]    resources:      requests: {cpu: 10m, memory: 32Mi}      limits: {memory: 64Mi}EOFpod/hungry created$sleep 30; kubectl get pod hungry -n tshNAME     READY   STATUS      RESTARTS   AGEhungry   0/1     OOMKilled   0          30s

   Expected resultOOMKilled in the STATUS column. With restartPolicy: Always you would more often catch it as CrashLoopBackOff with restarts climbing, and would need lastState to find the reason.

   Success conditionThe Pod's status is OOMKilled.

2. 2

   137, and where the number comes from

   OOMKilled exit=137, and the 137 is not arbitrary.

   Exit codes for signalled processes are 128 + signal number. SIGKILL is signal 9. 128 + 9 = 137. So 137 always means "killed with SIGKILL", and in a container that is nearly always the OOM killer.

   The same arithmetic explains its neighbours: 143 is 128 + 15, SIGTERM, which is a normal shutdown; 139 is 128 + 11, SIGSEGV.

   That matters when the reason field is unhelpful. On some runtime and kernel combinations a container killed for memory reports Error with exit 137 rather than the tidy OOMKilled. Exit 137 is the reliable indicator; the reason string is the convenience.

   describe shows the timing, and it is worth noticing:

   Started:   09:27:13
   Finished:  09:27:13

   The same second. The container allocated past its limit and was killed effectively instantly. Compare that against an application that runs for hours and is then OOMKilled, which points at a slow leak rather than a limit set below what startup needs. The gap between Started and Finished is a genuine diagnostic: seconds means the limit is simply too small, days means something is growing.

   Finally, the limit and request side by side: limit=64Mi request=32Mi. Both are worth reading, because only the limit causes the kill. The request affects scheduling and QoS class, never enforcement.

   $kubectl get pod hungry -n tsh -o jsonpath="{.status.containerStatuses[0].state.terminated.reason}{\" exit=\"}{.status.containerStatuses[0].state.terminated.exitCode}{\"\n\"}"OOMKilled exit=137$kubectl describe pod hungry -n tsh | grep -A5 'Last State\|State:' | head -12    State:          Terminated      Reason:       OOMKilled      Exit Code:    137      Started:      Fri, 21 Aug 2026 09:27:13 +0000      Finished:     Fri, 21 Aug 2026 09:27:13 +0000    Ready:          False$kubectl get pod hungry -n tsh -o jsonpath="limit={.spec.containers[0].resources.limits.memory}{\" request=\"}{.spec.containers[0].resources.requests.memory}{\"\n\"}"limit=64Mi request=32Mi

   Expected resultReason, exit code, and a Started/Finished pair in the same second. Note this appears under State: rather than Last State: because restartPolicy: Never means there is no next container; with Always you would read the same fields under lastState.

   Success conditionExit code 137 with reason OOMKilled.

3. 3

   Which failure is which

   The overview table from the same capture, because OOMKilled's place among the others is the useful context.

   badimage   Pending   ImagePullBackOff   <none>      0
   crasher    Running   <none>             Error       4
   hungry     Failed    <none>             OOMKilled   0
   noauth     Pending   ImagePullBackOff   <none>      0
   nonode     Pending   <none>             <none>      <none>
   toobig     Pending   <none>             <none>      <none>

   Four distinct shapes, and each tells you where to look:

   • Pending + a waiting reason - the Pod has a node and the container will not start. Image or mount problem.
   • Running + a terminated reason + restarts - the container starts and dies. Application problem; read the exit code.
   • Failed + OOMKilled - killed for memory, and not coming back because the restart policy says so.
   • Pending + nothing at all - never scheduled. containerStatuses does not exist yet, which is why every column reads .

   That last row is the one worth committing to memory. An unschedulable Pod has no container status, so any script or query reaching into .status.containerStatuses returns nothing and tells you nothing. For those, the information is in .status.conditions and the events, which the Pending guide covers.

   hungry reporting phase Failed is also the correct behaviour to expect here: with restartPolicy: Never a terminal container puts the Pod into a terminal phase. Under Always the phase would be Running, exactly as crasher's is, and just as misleading.

   $kubectl get pods -n tshNAME       READY   STATUS             RESTARTS      AGEbadimage   0/1     ImagePullBackOff   0             2m38scrasher    0/1     Error              4 (58s ago)   103shungry     0/1     OOMKilled          0             57snoauth     0/1     ImagePullBackOff   0             2m13snonode     0/1     Pending            0             12stoobig     0/1     Pending            0             27s$kubectl get pods -n tsh -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,REASON:.status.containerStatuses[0].state.waiting.reason,TERM:.status.containerStatuses[0].state.terminated.reason,RESTARTS:.status.containerStatuses[0].restartCount --no-headersbadimage   Pending   ImagePullBackOff   <none>      0crasher    Running   <none>             Error       4hungry     Failed    <none>             OOMKilled   0noauth     Pending   ImagePullBackOff   <none>      0nonode     Pending   <none>             <none>      <none>toobig     Pending   <none>             <none>      <none>

   Expected resultSix deliberately broken Pods, four distinct failure shapes. READY 0/1 is common to all of them and is the column worth scanning first on a real cluster.

   Success conditionYou can name the failure class from the phase and reason columns alone.

Troubleshooting

• A container is OOMKilled and its memory usage looked fine in monitoring.

  Why: Monitoring samples periodically; an allocation spike between samples is invisible. The kill happens at the instant the limit is crossed.

  Fix:Do not trust averages for this. kubectl top pod is also a sample, not a peak. Set the limit from the application's genuine worst case, which for a JVM means heap plus metaspace plus thread stacks plus native buffers, not just -Xmx. The Started-to-Finished gap tells you which kind of problem it is: seconds means the limit is too small for startup, days means a leak.

• The reason says Error but the exit code is 137.

  Why: Some runtime and kernel combinations do not label the kill. The exit code is the reliable field.

  Fix:Treat 137 as OOM until proved otherwise. Confirm on the node: dmesg -T | grep -i 'killed process' shows the kernel's own OOM record with the process name and the cgroup, which is definitive. Do this before raising the limit, because 137 can also come from something else sending SIGKILL.

• Raising the limit did not stop the kills.

  Why: Either the application genuinely leaks, or something inside it sizes itself from the node's memory rather than the cgroup limit.

  Fix:The second case is common with older runtimes that read /proc/meminfo and see the whole node. Modern JVMs and .NET are container-aware; older ones need explicit flags. Check what the process thinks it has: kubectl exec <pod> -- cat /sys/fs/cgroup/memory.max is the limit it should be respecting. If usage climbs steadily under constant load, it is a leak and no limit will fix it.

• Other Pods on the node were evicted rather than the greedy one.

  Why: Node-level memory pressure triggers eviction, which is a different mechanism from a cgroup OOM kill and chooses victims by QoS class. BestEffort Pods go first, then Burstable that exceed their requests.

  Fix:Distinguish the two: an OOM kill shows OOMKilled on the container, while an eviction shows the Pod with a Failed phase and reason Evicted. To protect a workload, give it requests equal to its limits so it is Guaranteed, which is the last class to be evicted. The QoS guide covers the classes.

• A container writing to a volume is OOMKilled and you do not see why.

  Why: An emptyDir with medium: Memory, or /dev/shm, is a tmpfs. Files written there count against the memory limit, as this guide's own Pod demonstrates.

  Fix:Check for tmpfs mounts: kubectl exec <pod> -- df -h | grep tmpfs. Either budget the limit to cover the data or move it to a disk-backed volume. /dev/shm defaults to a small size but still draws on the container's memory budget.

• You need the application's last words and there are none.

  Why: SIGKILL cannot be handled. No flush, no shutdown hook, no final log line.

  Fix:Structurally unavoidable, so instrument earlier: log memory-relevant state periodically rather than at exit. A preStop hook does not help either, since it only runs for a graceful termination. If you need a heap dump at the moment of failure, configure the runtime to write one on its own OOM rather than relying on the kernel's.

Official sources

• Assign memory resources to containers
• Resource management for Pods and containers
• Quality of Service for Pods
• Node-pressure eviction
• Debug Pods