kubectl debug and Ephemeral Containers

Troubleshooting / Guide 78 of 103 / Advanced

• Kubernetes1.36.4
• Runtimecontainerd 2.2.6
• Cluster4 nodes
• CNICalico v3.32.1
• TimeAbout 35 min
• Reviewed21 August 2026

Written against the versions above. Ephemeral containers cannot be removed once added. They live until the Pod does.

Before you start

• The container logs guide, and comfort with kubectl exec.
• RBAC allowing pods/ephemeralcontainers (patch) and pods/exec (create). Cluster-admin has both.
• The RBAC guide, because kubectl debug node/ is close to root on that node and should be treated as such.

1. 1

   Exec fails when there is no shell to exec

   The Pod runs registry.k8s.io/pause:3.10, a container that is a single static binary and nothing else. It is a fair stand-in for a distroless or FROM scratch image, which is increasingly what production containers look like.

   Both attempts fail, and the errors are precise:

   exec: "sh": executable file not found in $PATH
   exec: "/bin/ls": stat /bin/ls: no such file or directory

   There is no sh. There is no /bin/ls. There is no /bin. Nothing is broken: the image was built to contain one binary, and that is a deliberate security property, since an attacker who reaches the container has no tools either.

   So the debugging advice everyone learns first, kubectl exec -it -- sh, does not work on a growing share of real workloads. The instinct to rebuild the image with a shell "just for debugging" gives up the security property permanently to solve a problem that has a proper answer.

   Note also the phrasing of the first error: not found in $PATH. That distinguishes it from a shell that exists but cannot run, which would be a permissions or architecture error instead.

   $kubectl run distroless -n dbg --image=registry.k8s.io/pause:3.10 --restart=Neverpod/distroless created$kubectl wait --for=condition=Ready pod/distroless -n dbg --timeout=180spod/distroless condition met$kubectl exec distroless -n dbg -- sh -c 'echo hello' 2>&1 | tail -2error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "4e310ff67fc13a0121569cf3ad4e6fb68174f6c0fb0b8f552eb40ad8a113f706": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH$kubectl exec distroless -n dbg -- /bin/ls 2>&1 | tail -2error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "3d35aa6f0ecf431b46a6c76f14e737a449ed32ba7c946c38578d2240e50f23aa": OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/ls": stat /bin/ls: no such file or directory

   Expected resultTwo failures from the runtime, not from Kubernetes. OCI runtime exec failed tells you the request reached containerd and containerd could not honour it, which rules out RBAC and the kubelet. Note the two messages differ: a bare name that is not in $PATH gives executable file not found, while an absolute path that does not exist gives stat ...: no such file or directory. Both mean the same thing here, and the distinction matters when the file does exist but cannot run, which reports a permission or format error instead.

   Success conditionkubectl exec fails with executable file not found.

2. 2

   An ephemeral container, sharing the target's namespaces

   kubectl debug adds a new container to a running Pod, without restarting it. --target=distroless is the important flag: it makes the new container share the target's process namespace.

   -q suppresses the informational output, and -- sleep 300 gives it something to do so it stays up long enough to exec into. Without a command it runs the image's default, which for netshoot is an interactive shell that exits immediately in a non-interactive session.

   The proof that the sharing worked is ps aux:

       1 65535     0:00 /pause
      20 root      0:00 sleep 300

   PID 1 is /pause - the target container's process, visible from inside the debug container. That is the whole point: you can inspect the real process, read /proc/1/environ, check what it has open, send it signals. All the tools come from the netshoot image; all the processes come from the workload.

   The network namespace is shared too. 10.244.93.62/32 on eth0 is the Pod's address, so curl localhost:8080 from here reaches the target's listener, and tcpdump sees the target's traffic. That is what makes this the right tool for network debugging in a container with no networking tools.

   What is not shared is the filesystem. The debug container has netshoot's root filesystem, not the target's. To read the target's files, go through /proc/1/root/ which the shared process namespace makes available.

   Two details in the status worth noting. The ephemeral container reports ready=false, and always will: ephemeral containers have no probes, so readiness is never established. And the Pod still shows 1/1 Running rather than 2/2, because ephemeral containers do not count toward the READY column or affect the Pod's readiness. A Pod is not made unready by attaching a debugger to it.

   $kubectl debug distroless -n dbg --image=nicolaka/netshoot:latest --target=distroless -q -- sleep 300$sleep 20; kubectl get pod distroless -n dbg -o jsonpath="{range .status.ephemeralContainerStatuses[*]}{.name}{\" ready=\"}{.ready}{\"\n\"}{end}"debugger-wlwzk ready=false$kubectl exec distroless -n dbg -c debugger-wlwzk -- ps aux | head -5PID   USER     TIME  COMMAND    1 65535     0:00 /pause   20 root      0:00 sleep 300   27 root      0:00 ps aux$kubectl exec distroless -n dbg -c debugger-wlwzk -- ip -4 addr show eth0 | grep inet    inet 10.244.93.62/32 scope global eth0$kubectl get pod distroless -n dbg -o jsonpath="{range .spec.ephemeralContainers[*]}{.name}{\" image=\"}{.image}{\" target=\"}{.targetContainerName}{\"\n\"}{end}"debugger-wlwzk image=nicolaka/netshoot:latest target=distroless$kubectl get pod distroless -n dbg --no-headers | awk '{print $1, $2, $3}'distroless 1/1 Running

   Expected result/pause at PID 1 inside the debug container, the Pod's own IP on eth0, and the Pod still reporting 1/1. The generated name debugger-wlwzk is random; read it from the status rather than guessing.

   Success conditionps aux in the ephemeral container shows the target's process at PID 1.

3. 3

   A copy of the Pod you can break

   --copy-to takes the other approach: instead of adding to the running Pod, it creates a new Pod from the same spec with your changes applied.

   Here --container=web --image=netshoot replaces just that container's image. Read the result:

   web nicolaka/netshoot:latest
   sidecar busybox:1.36

   The web container is now netshoot and sidecar is untouched. The copy keeps the original's volumes, environment, ServiceAccount and labels, so it runs in the same context as the real thing while you take it apart.

   When to use which:

   • --target for a live problem. Same Pod, same processes, same network, nothing restarted. This is the only option when the state you need to inspect is in the running process.
   • --copy-to when you need to change something. A different image, a different command, an added environment variable. The original keeps serving.

   A caveat on the copy worth being explicit about: because it has the same labels, a Service in front of the original will send traffic to the copy too. If the copy is broken or instrumented, production traffic reaches it. Either strip the labels afterwards, or accept it knowingly. --copy-to with --replace deletes the original instead, which is a different and much more invasive choice.

   The dig result confirms the copy is a fully functioning Pod: cluster DNS resolves from it, so anything you test there behaves as the original would.

   $kubectl debug app -n dbg --copy-to=app-debug --container=web --image=nicolaka/netshoot:latest -q -- sleep 300$kubectl wait --for=condition=Ready pod/app-debug -n dbg --timeout=180spod/app-debug condition met$kubectl get pods -n dbg --no-headers | awk '{print $1, $2, $3}'app 2/2 Runningapp-debug 2/2 Runningdistroless 1/1 Running$kubectl exec app-debug -n dbg -c web -- dig +short kubernetes.default.svc.cluster.local10.96.0.1$kubectl get pod app-debug -n dbg -o jsonpath="{range .spec.containers[*]}{.name}{\" \"}{.image}{\"\n\"}{end}"web nicolaka/netshoot:latestsidecar busybox:1.36

   Expected resultBoth Pods running, one container swapped in the copy and the other left alone. The original app is untouched throughout.

   Success conditionapp-debug exists with a replaced image and the original still running.

4. 4

   A root shell on a node

   kubectl debug node/ is a different thing again, and the most powerful command in this guide. It creates a Pod on that node with the node's filesystem mounted at /host.

   The listing proves it:

   kubelet.conf
   manifests
   pki

   That is /etc/kubernetes on cka1001-node01, read from inside a container. pki holds the node's private keys.

   The Pod's spec says the rest:

   hostNetwork=true hostPID=true node=cka1001-node01

   Host network, host PID namespace, and the whole root filesystem mounted. That is effectively a root shell on the node, reached through the Kubernetes API. Concretely, whoever can run this can read every Secret's decrypted content from /host/var/lib/kubelet, read the control plane's certificates if run against a control plane node, and write to /host/etc/kubernetes/manifests to have the kubelet start any container they like as root.

   So two things follow.

   Operationally it is superb, and it is the right tool when a node is misbehaving and you cannot SSH to it: journalctl through /host, crictl, disk usage, dmesg for OOM records.

   From a security point of view, create pods in any namespace is close to cluster-admin, and this command is the clearest demonstration of why. Granting someone the ability to create Pods on a cluster grants them this, whether or not they know the command exists. Treat node access as the privilege it is when writing RBAC.

   The debug Pod is a normal Pod and does not clean itself up; delete it, or the namespace, when finished.

   $kubectl debug node/cka1001-node01 -n dbg --image=busybox:1.36 -q -- sleep 120 2>&1 | tail -2$sleep 15; kubectl get pods -n dbg --no-headers | grep node-debugger | awk '{print $1, $3}'node-debugger-cka1001-node01-rrr85 Running$kubectl exec node-debugger-cka1001-node01-rrr85 -n dbg -- ls /host/etc/kubernetes 2>&1 | head -5kubelet.confmanifestspki$kubectl get pod node-debugger-cka1001-node01-rrr85 -n dbg -o jsonpath="hostNetwork={.spec.hostNetwork}{\" hostPID=\"}{.spec.hostPID}{\" node=\"}{.spec.nodeName}{\"\n\"}"hostNetwork=true hostPID=true node=cka1001-node01$kubectl delete ns dbg --wait=falsenamespace "dbg" deleted

   Expected resultThe node's /etc/kubernetes contents, and a spec with hostNetwork and hostPID both true. The Pod name embeds the node name plus a random suffix.

   Success conditionYou can list the node's filesystem through /host.

Troubleshooting

• kubectl exec fails with executable file not found in $PATH.

  Why: The image has no shell. Distroless, scratch, and single-binary images have nothing to exec.

  Fix:Use kubectl debug <pod> --image=nicolaka/netshoot --target=<container> -it -- sh. Do not rebuild the image with a shell to make exec work; that permanently removes the property the image was built for.

• kubectl debug reports that ephemeral containers are not enabled or the subresource is missing.

  Why: An older cluster, or RBAC without pods/ephemeralcontainers.

  Fix:Ephemeral containers are stable from 1.25, so on a modern cluster it is RBAC: the verb needed is patch on pods/ephemeralcontainers, which is not implied by create pods. Check with kubectl auth can-i patch pods/ephemeralcontainers. Fall back to --copy-to, which only needs create pods.

• The ephemeral container exits immediately.

  Why: It ran the image's default command with no TTY. An interactive shell with nothing attached exits at once.

  Fix:Give it something to do: -- sleep 3600, then exec in separately. Or use -it to attach immediately, which is the interactive form. -q only suppresses kubectl's own output and does not affect this.

• You attached the wrong image or made a mistake and want to remove the ephemeral container.

  Why: Ephemeral containers cannot be removed. The API allows adding only.

  Fix:Add another with the right image; several can coexist. To get back to a clean Pod, delete and recreate it, which for a Deployment-managed Pod means just deleting it. This is a good reason to reach for --copy-to when experimenting.

• A --copy-to copy started receiving production traffic.

  Why: The copy inherits the original's labels, so any Service selecting the original selects the copy.

  Fix:Change the labels immediately: kubectl label pod <copy> app- --overwrite removes the selector match. Check what would select it before creating the copy, and prefer --target on a live Pod when you do not actually need to change anything.

• The debug container cannot see the target's files.

  Why: --target shares the process and network namespaces, not the filesystem. Each container keeps its own root.

  Fix:Reach it through the shared process namespace: ls /proc/1/root/ is the target's filesystem, and cat /proc/1/environ | tr '\0' '\n' gives you its environment. If you need a writable view or the target has exited, --copy-to with the same volumes is the alternative.

• You want to restrict who can run kubectl debug node/.

  Why: It needs only create pods, because it is an ordinary Pod with hostPID, hostNetwork and a hostPath mount.

  Fix:RBAC cannot distinguish it from any other Pod creation, so the control is admission: Pod Security Admission at baseline or restricted forbids hostPath and host namespaces, which blocks it. Recognise that in a namespace without such a policy, create pods is equivalent to root on any node.

Official sources

• Debug running Pods
• Ephemeral containers
• kubectl debug reference
• Share process namespace between containers
• Pod Security Standards