Debugging Containers with Ephemeral Containers

Troubleshooting and Debugging / Guide 28 of 46 / Beginner

• Kubernetes1.36.4
• Cluster4 nodes
• Runtimecontainerd 2.2.6
• CNICalico v3.32.1
• TimeAbout 16 min
• Reviewed22 August 2026

Written against the versions above. Ephemeral containers, and so `kubectl debug`, have been stable since 1.25 and are available on any current cluster. The generated debugger name is random - yours will differ from the one shown.

Before you start

• A cluster and kubectl. kubectl debug needs the EphemeralContainers feature, which is on by default from 1.25.
• This session uses a dbg namespace with a two-container Pod named app (web running nginx, sidecar printing a tick every five seconds) and a single-container Pod named distroless running registry.k8s.io/pause:3.10, an image containing essentially nothing.

1. 1

   kubectl logs already chose a container for you

   The Pod has two containers and is 2/2 Running. Ask for its logs with no other flags and you get nginx - the first container - with no indication that a choice was made.

   kubectl exec is more honest about it:

   Defaulted container "web" out of: web, sidecar

   Both commands default to the first container; only one tells you. That is the single most common way people conclude "there are no logs" while looking at the wrong container.

   --all-containers with --prefix removes the ambiguity entirely, and the prefix names the Pod and the container:

   [pod/app/web] ... start worker process 30
   [pod/app/sidecar] sidecar tick 1

   $kubectl get pod app -n dbgNAME   READY   STATUS    RESTARTS   AGEapp    2/2     Running   0          1s$kubectl logs app -n dbg 2>&1 | tail -32026/08/21 09:37:57 [notice] 1#1: start worker processes2026/08/21 09:37:57 [notice] 1#1: start worker process 292026/08/21 09:37:57 [notice] 1#1: start worker process 30$kubectl exec app -n dbg -- hostnameDefaulted container "web" out of: web, sidecarapp$kubectl logs app -n dbg --all-containers=true --prefix=true --tail=4[pod/app/web] 2026/08/21 09:37:57 [notice] 1#1: start worker process 29[pod/app/web] 2026/08/21 09:37:57 [notice] 1#1: start worker process 30[pod/app/sidecar] sidecar tick 1

   Expected resultDefault logs come from the first container, and --all-containers labels both.

   Success conditionYou know which container you have been reading.

2. 2

   The four flags that make logs usable

   kubectl logs with no flags on a busy container is unreadable. Four flags do almost all the work:

   --since - a time window rather than a line count, which is what you actually want when correlating with an incident:

   sidecar tick 2
   sidecar tick 3

   --timestamps - because container logs carry no clock of their own unless the application put one there:

   2026-08-21T09:38:02.395439433Z sidecar tick 2

   -l - a label selector instead of a Pod name, so you read every replica at once. With --prefix you can tell them apart. This is the one that scales: you rarely know which of five Pods served the bad request.

   --tail - a bounded number of lines, which is worth making a habit for any container that has been running for days.

   There is also --previous, for the logs of a container that has already died and been replaced - the single most useful flag when something is CrashLooping, and one that only works while the dead container has not yet been garbage collected.

   $sleep 12; kubectl logs app -n dbg -c sidecar --since=10ssidecar tick 2sidecar tick 3$kubectl logs app -n dbg -c sidecar --timestamps=true --tail=22026-08-21T09:38:02.395439433Z sidecar tick 22026-08-21T09:38:07.395575353Z sidecar tick 3$kubectl logs -n dbg -l app=demo -c sidecar --tail=2 --prefix=true[pod/app/sidecar] sidecar tick 2[pod/app/sidecar] sidecar tick 3

   Expected resultA time-windowed read, timestamped lines, and a label-selected read with prefixes.

   Success conditionYou can pull the last ten seconds from every replica of a Deployment.

3. 3

   exec works until the image has nothing to exec

   kubectl exec runs a process *inside* an existing container, which means the binary has to already be in the image. On nginx that is fine:

   200

   That curl is worth pausing on - it ran inside the container against 127.0.0.1, which proves the app is serving on its own loopback. If that returns 200 and the Service does not work, the problem is not the application.

   Now the same idea against a minimal image. registry.k8s.io/pause:3.10 contains a single static binary and no userland at all:

   OCI runtime exec failed: exec failed: unable to start container process:
   exec: "sh": executable file not found in $PATH

   exec: "/bin/ls": stat /bin/ls: no such file or directory

   Not a permissions problem and not a Kubernetes problem - there is genuinely no shell in there. Distroless and scratch images are built this way on purpose: fewer binaries is a smaller attack surface. The consequence is that the usual kubectl exec -it ... sh is simply unavailable on exactly the images security review asks you to use.

   $kubectl exec app -n dbg -c web -- curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1200$kubectl exec distroless -n dbg -- sh -c 'echo hello' 2>&1 | tail -2error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "4e310ff67fc13a0121569cf3ad4e6fb68174f6c0fb0b8f552eb40ad8a113f706": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH$kubectl exec distroless -n dbg -- /bin/ls 2>&1 | tail -2error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "3d35aa6f0ecf431b46a6c76f14e737a449ed32ba7c946c38578d2240e50f23aa": OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/ls": stat /bin/ls: no such file or directory

   Expected resultexec succeeds on nginx and fails twice on the minimal image, for lack of a binary.

   Success conditionYou can tell "no shell in the image" from a Kubernetes error.

4. 4

   kubectl debug brings its own tools

   The answer is not to put a shell in the production image. It is to add a container to the running Pod that has one:

   kubectl debug distroless --image=nicolaka/netshoot:latest --target=distroless -- sleep 300

   That creates an ephemeral container - a container added to a Pod after it is running, which the Pod's spec did not declare and which no controller will recreate. It appears in its own status list, not among the normal containers:

   debugger-wlwzk ready=false

   The name is generated; note it, because you need it to exec in. Then, from inside the debugger, look at the process table:

   PID   USER     TIME  COMMAND
       1 65535     0:00 /pause
      20 root      0:00 sleep 300
      27 root      0:00 ps aux

   PID 1 is /pause - the target container's process. --target joined the debugger to that container's process namespace, so a netshoot image full of tools is now looking directly at a container that has none of them. The workload was never modified and never restarted.

   Two things worth knowing about the shape of this. An ephemeral container cannot be removed - it lives until the Pod does, which is a reason to delete the Pod when you are finished. And ready=false is normal; ephemeral containers are excluded from the Pod's readiness, precisely so attaching a debugger cannot take a Service endpoint out of rotation.

   $kubectl debug distroless -n dbg --image=nicolaka/netshoot:latest --target=distroless -q -- sleep 300$sleep 20; kubectl get pod distroless -n dbg -o jsonpath="{range .status.ephemeralContainerStatuses[*]}{.name}{\" ready=\"}{.ready}{\"\n\"}{end}"debugger-wlwzk ready=false$kubectl exec distroless -n dbg -c debugger-wlwzk -- ps aux | head -5PID   USER     TIME  COMMAND    1 65535     0:00 /pause   20 root      0:00 sleep 300   27 root      0:00 ps aux

   Expected resultAn ephemeral container attached to the Pod, seeing the target container's PID 1.

   Success conditionYou debugged a container that has no shell, without changing its image.

Troubleshooting

• "There are no logs" from a Pod that is clearly doing something.

  Why: You are reading the first container in a multi-container Pod. kubectl logs defaults silently; only kubectl exec announces the default.

  Fix:kubectl logs <pod> --all-containers=true --prefix=true, or name the container with -c.

• kubectl exec fails with "executable file not found in $PATH".

  Why: The image has no shell. Distroless, scratch and pause images ship no userland by design.

  Fix:kubectl debug <pod> --image=nicolaka/netshoot --target=<container> -- sleep 300, then exec into the generated debugger container.

• kubectl logs --previous says it cannot retrieve the logs.

  Why: The dead container has already been garbage collected by the kubelet. Its logs are gone.

  Fix:Nothing retroactively. Ship logs off the node. A collector running as a DaemonSet reads the node's container log files and forwards them, which keeps them past container garbage collection.

• An ephemeral container will not go away.

  Why: By design - ephemeral containers cannot be removed from a running Pod.

  Fix:Delete the Pod when you are done. If it is managed by a controller a fresh one replaces it without the debugger.

Official sources

• Kubernetes documentation - Debug Running Pods
• Kubernetes documentation - Ephemeral Containers
• Kubernetes documentation - kubectl logs reference
• Kubernetes documentation - kubectl debug reference