Job Completions, Parallelism and backoffLimit

Application Design and Build / Guide 10 of 44 / Beginner

• Kubernetes1.36.4
• Runtimecontainerd 2.2.6
• CNICalico v3.32.1
• TimeAbout 14 min
• Reviewed23 August 2026

Written against the versions above. With `restartPolicy: Never` each attempt is a NEW Pod, and the failed Pods are kept so you can read their logs. With `restartPolicy: OnFailure` the same Pod's container is restarted in place and you get one Pod with a restart count instead - the same retry budget, counted differently.

Before you start

• A cluster and kubectl.
• The session creates namespace ckad-job, a Job batch with four completions two at a time, and a Job doomed whose container always exits 1.

1. 1

   Four completions, two at a time

   completions: 4
   parallelism: 2

   Four successful runs are required and two may run at once, so the controller keeps two Pods alive until four have succeeded:

   NAME    STATUS     COMPLETIONS   DURATION   AGE
   batch   Complete   4/4           ...

   The Pods spread across the workers, and each logged its own hostname - four distinct Pod names, which is what completions: 4 means. Set completions without parallelism and they run one at a time; set parallelism without completions and any single success finishes the Job.

   $kubectl create namespace ckad-jobnamespace/ckad-job created$kubectl -n ckad-job wait --for=condition=Complete job/batch --timeout=180sjob.batch/batch condition met$kubectl -n ckad-job get job batchNAME    STATUS     COMPLETIONS   DURATION   AGEbatch   Complete   4/4           12s        12s$kubectl -n ckad-job get pods -l batch.kubernetes.io/job-name=batch -o 'custom-columns=POD:.metadata.name,PHASE:.status.phase,NODE:.spec.nodeName'POD           PHASE       NODEbatch-dphlg   Succeeded   cka1001-node01batch-qszkv   Succeeded   cka1001-node01batch-tvbj7   Succeeded   cka1001-node01batch-wgwjz   Succeeded   cka1001-node01$kubectl -n ckad-job logs -l batch.kubernetes.io/job-name=batch --tail=1done on batch-dphlgdone on batch-qszkvdone on batch-tvbj7done on batch-wgwjz

   Expected resultComplete 4/4 and four Pods across the workers.

   Success conditionYou can express "run this N times, M at a time" correctly.

2. 2

   One that gives up

   backoffLimit: 2 with a container that always exits 1. After forty-five seconds:

   NAME     STATUS   COMPLETIONS   DURATION   AGE
   doomed   Failed   0/1           45s        45s

   and three Pods, not two:

   POD            PHASE    EXIT
   doomed-ddzh4   Failed   1
   doomed-jh556   Failed   1
   doomed-r5n2s   Failed   1

   This is the detail worth carrying: backoffLimit counts retries after the first attempt. One initial run plus two retries is three Pods. Set it to 0 and you get exactly one attempt.

   The retries are also spaced out - 10s, 20s, 40s, doubling to a six-minute cap - which is why a Job with a high backoffLimit can take a very long time to declare failure.

   $sleep 45; kubectl -n ckad-job get job doomedNAME     STATUS   COMPLETIONS   DURATION   AGEdoomed   Failed   0/1           45s        45s$kubectl -n ckad-job get pods -l batch.kubernetes.io/job-name=doomed -o 'custom-columns=POD:.metadata.name,PHASE:.status.phase,EXIT:.status.containerStatuses[0].state.terminated.exitCode'POD            PHASE    EXITdoomed-ddzh4   Failed   1doomed-jh556   Failed   1doomed-r5n2s   Failed   1

   Expected resultFailed, and three Pods each with exit code 1.

   Success conditionYou can predict how many attempts a backoffLimit produces.

3. 3

   Reading why it stopped

   The Job's own conditions say it in one line:

   FAILED   CONDITION              REASON
   3        FailureTarget,Failed   BackoffLimitExceeded,BackoffLimitExceeded

   BackoffLimitExceeded - the Job did not fail because of what the container did, it failed because it ran out of retries. That distinction matters when you are deciding whether to raise the limit or fix the code.

   And because restartPolicy: Never keeps every failed Pod, the evidence is still there:

   attempt on doomed-ddzh4
   attempt on doomed-jh556
   attempt on doomed-r5n2s

   Three Pods, three sets of logs, one label selector. On a Job that failed hours ago this is usually the fastest way in.

   $kubectl -n ckad-job get job doomed -o 'custom-columns=FAILED:.status.failed,CONDITION:.status.conditions[*].type,REASON:.status.conditions[*].reason'FAILED   CONDITION              REASON3        FailureTarget,Failed   BackoffLimitExceeded,BackoffLimitExceeded$kubectl -n ckad-job logs -l batch.kubernetes.io/job-name=doomed --tail=1attempt on doomed-ddzh4attempt on doomed-jh556attempt on doomed-r5n2s

   Expected resultBackoffLimitExceeded and the logs of all three attempts.

   Success conditionYou can tell "the code is broken" from "the retry budget ran out".

4. 4

   restartPolicy decides what you are counting

   Never

   3

   With Never, each attempt is a fresh Pod and they accumulate - three attempts, three Pods. With OnFailure, the kubelet restarts the container inside the same Pod and you see one Pod with RESTARTS 3 instead.

   Which to choose:

   • Never when you want the logs of every attempt separately, which is most of the time during development.
   • OnFailure when the retries are expected and you do not want dozens of dead Pods.

   A Job's template may only use those two. restartPolicy: Always is rejected outright - it would mean the Pod can never finish, which is the same problem guide 22 runs into from the other direction.

   Finally, nothing cleans up finished Jobs by default. ttlSecondsAfterFinished on the Job deletes it and its Pods a set time after it finishes, and on a cluster that runs batch work it is worth setting.

   $kubectl -n ckad-job get job doomed -o jsonpath='{.spec.template.spec.restartPolicy}'Never$kubectl -n ckad-job get pods -l batch.kubernetes.io/job-name=doomed --no-headers | wc -l3$kubectl delete namespace ckad-job --wait=falsenamespace "ckad-job" deleted

   Expected resultNever, and a count of three Pods.

   Success conditionYou can choose a restartPolicy for the debugging you expect to do.

Troubleshooting

• A Job made more Pods than backoffLimit.

  Why: The limit counts retries, not total attempts.

  Fix:Expected: attempts = backoffLimit + 1. Use backoffLimit: 0 for exactly one try.

• A failing Job takes a very long time to be marked Failed.

  Why: Retries back off exponentially, up to six minutes apart.

  Fix:Lower backoffLimit, or set activeDeadlineSeconds to cap the whole Job.

• The failed Pods are gone and you cannot read the logs.

  Why: restartPolicy: OnFailure restarts in place; or a TTL deleted them.

  Fix:Use restartPolicy: Never while debugging. Check ttlSecondsAfterFinished.

• restartPolicy: Always is rejected on a Job.

  Why: A Pod that always restarts can never complete.

  Fix:Use Never or OnFailure.

Official sources

• Jobs
• Job - handling Pod and container failures
• Automatic cleanup for finished Jobs