CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Application Developer

Sidecar Containers in Jobs

A Job is complete when its Pod's containers have all terminated. A sidecar by definition never terminates, so a Job with an ordinary sidecar runs forever with the actual work finished in the first second. The native sidecar - an init container with `restartPolicy: Always` - exists precisely to fix this, and both Jobs are run here side by side so the difference is one table.

Application Design and Build Guide 9 of 44 Intermediate

Written against the versions above. Native sidecars went stable in Kubernetes 1.33 and are used here on 1.36. A container in `initContainers` with `restartPolicy: Always` starts before the application containers, keeps running alongside them, and is excluded from the Job's completion calculation. On older clusters the workaround was to have the sidecar watch for a marker file and exit itself.

Two Jobs on the worker nodes. The behaviour shown is a property of the Job controller, not of any node.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. A Job with an ordinary sidecar

    Two containers: work writes a file and exits, helper loops forever. Thirty seconds later:

    NAME     STATUS    COMPLETIONS   DURATION   AGE
broken   Running   0/1           30s        30s

    0/1, still Running. But the work is done - it logged its line in the first second:

    finished

    The Pod confirms it: one container ready, one not.

    POD            PHASE     READY
broken-hfdvj   Running   true,false

    The Job will never complete. Its Pod cannot reach a terminal phase while a container is still running, and the helper has no reason to stop. With no activeDeadlineSeconds this runs until somebody notices.

    bash Example session 
    kubectl create namespace ckad-sidenamespace/ckad-side createdsleep 30; kubectl -n ckad-side get job brokenNAME     STATUS    COMPLETIONS   DURATION   AGEbroken   Running   0/1           30s        30skubectl -n ckad-side get pods -l batch.kubernetes.io/job-name=broken -o 'custom-columns=POD:.metadata.name,PHASE:.status.phase,READY:.status.containerStatuses[*].ready'POD            PHASE     READYbroken-hfdvj   Running   true,falsekubectl -n ckad-side logs -l batch.kubernetes.io/job-name=broken -c workfinished

    Expected resultRunning 0/1 with the work container's output already present.

    Success conditionYou have reproduced a Job that is finished and does not know it.

  2. The same Job with a native sidecar

    The only change: helper moves from containers to initContainers, and gains restartPolicy: Always.

    NAME    STATUS     COMPLETIONS   DURATION   AGE
fixed   Complete   1/1           34s        34s
    POD           PHASE
fixed-kk2fz   Succeeded

    Complete. The helper still ran - it started *before* the work container, which is the other half of what native sidecars give you, and is why they suit a proxy or a log shipper that has to be up before the app produces anything. But it is not counted, so when work exited the Pod succeeded and Kubernetes shut the sidecar down.

    That ordering guarantee is the reason this is not just a flag on a normal container. An ordinary sidecar starts *at the same time* as the app and may miss its first output.

    bash Example session 
    kubectl -n ckad-side wait --for=condition=Complete job/fixed --timeout=120sjob.batch/fixed condition metkubectl -n ckad-side get job fixedNAME    STATUS     COMPLETIONS   DURATION   AGEfixed   Complete   1/1           34s        34skubectl -n ckad-side get pods -l batch.kubernetes.io/job-name=fixed -o 'custom-columns=POD:.metadata.name,PHASE:.status.phase'POD           PHASEfixed-kk2fz   Succeeded

    Expected resultComplete 1/1 and a Succeeded Pod.

    Success conditionYou fixed a hanging Job by moving one container in the manifest.

  3. Side by side

    JOB      COMPLETIONS   ACTIVE
broken   <none>        1
fixed    1             <none>

    One line each. broken has no completions and an active Pod; fixed has its completion and nothing running.

    The rule worth keeping: any container that does not exit on its own belongs in initContainers with restartPolicy: Always, whenever the workload is a Job or a CronJob. In a Deployment it makes no difference to completion - nothing is ever expected to finish - but the start-ordering guarantee is still usually what you want.

    If you are stuck on an older cluster, the escape hatch is activeDeadlineSeconds on the Job, which kills the Pod after a set time. It marks the Job Failed rather than complete, so it is damage control and not a fix.

    bash Example session 
    kubectl -n ckad-side get jobs -o 'custom-columns=JOB:.metadata.name,COMPLETIONS:.status.succeeded,ACTIVE:.status.active'JOB      COMPLETIONS   ACTIVEbroken   <none>        1fixed    1             <none>kubectl -n ckad-side delete job brokenjob.batch "broken" deleted from ckad-side namespacekubectl delete namespace ckad-side --wait=falsenamespace "ckad-side" deleted

    Expected resultThe two Jobs in one table, one active and one complete.

    Success conditionYou can state the rule and show the evidence for it.

Troubleshooting

Official sources

← All CKAD guides