Events, describe and Object Status
Almost every Kubernetes failure explains itself in one of three places: an event, the Events section of describe, or a status condition. Knowing which of the three to read, and that events expire after an hour, is most of the skill.
Troubleshooting Guide 76 of 103 Beginner
- Kubernetes1.36.4
- Cluster4 nodes
- Runtimecontainerd 2.2.6
- TimeAbout 30 min
- Reviewed21 August 2026
Written against the versions above. Event reasons and condition names are stable across recent versions. The exact wording of messages is not, so read them for meaning rather than matching strings.
| Server Name | IP Address | OS | Roles | CPU | RAM | HDD |
|---|---|---|---|---|---|---|
| CKA1001 | 192.168.0.175 | Ubuntu 26.04 LTS | Control Plane Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE01 | 192.168.0.176 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE02 | 192.168.0.177 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE03 | 192.168.0.178 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
Before you start
- A running cluster and
kubectl. - The Pods and Deployments guides, since the examples are a Deployment and a bare Pod.
-
What an event actually is
An event is a short record written by a component about an object: the scheduler saying it could not place a Pod, the kubelet saying an image pull failed. Two broken workloads produce a readable trail:
Warning FailedScheduling toobig 0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory... Warning Failed bad-77dc8dbb75-krx5k Failed to pull image "nginx:does-not-exist-1234": rpc error: code = NotFound... Warning Failed bad-77dc8dbb75-krx5k Error: ErrImagePull Normal BackOff bad-77dc8dbb75-krx5k Back-off pulling image "nginx:does-not-exist-1234"Four fields carry the information.
Type is
NormalorWarning, and nothing else. That makes--field-selector type=Warningthe single highest-value filter in Kubernetes troubleshooting: it turns a wall of routine pull-and-start chatter into the short list of things that went wrong.Reason is a short CamelCase token from a fixed vocabulary:
FailedScheduling,ErrImagePull,BackOff,Unhealthy,OOMKilling,FailedMount. Reasons are what you filter and search on, because they are stable across versions in a way messages are not.involvedObject names what the event is about, and this is the field that trips people up. Events attach to the object a component was acting on, which is almost always the Pod, not the Deployment. Asking for events on a Deployment that is failing because its Pods cannot pull an image returns almost nothing useful, and the conclusion "there are no events" is wrong: they are on the ReplicaSet's Pods.
Message is free text, and the
FailedSchedulingone above is a good example of how much it can carry.0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memoryaccounts for every node in the cluster and says exactly why each was rejected. Scheduler messages of that shape are worth reading in full rather than skimming.Note
--sort-by=.lastTimestampon the command. Default event ordering is not chronological, which is confusing enough that the sort flag is worth typing every time.bash Example session kubectl get events -n tri --sort-by=.lastTimestamp -o custom-columns=TYPE:.type,REASON:.reason,OBJECT:.involvedObject.name,MESSAGE:.message --no-headers | tail -8Normal Pulled good-79f4495545-wbc6x Container image "nginx:1.29-alpine" already present on machine and can be accessed by the podNormal Created good-79f4495545-wbc6x Container createdWarning FailedScheduling toobig 0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory. no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.Normal Pulling bad-77dc8dbb75-krx5k Pulling image "nginx:does-not-exist-1234"Warning Failed bad-77dc8dbb75-krx5k Error: ErrImagePullWarning Failed bad-77dc8dbb75-krx5k Failed to pull image "nginx:does-not-exist-1234": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:does-not-exist-1234": failed to resolve image: docker.io/library/nginx:does-not-exist-1234: not foundWarning Failed bad-77dc8dbb75-krx5k Error: ImagePullBackOffNormal BackOff bad-77dc8dbb75-krx5k Back-off pulling image "nginx:does-not-exist-1234"Expected resultNormal events for the healthy Deployment mixed with warnings for the two broken workloads. The
goodPod'sPulledevent says "already present on machine", which is what a cached image looks like.Success conditionYou can list events with type, reason, object and message as separate columns.
-
Filter to warnings, and read the count
The filtered view is the one to reach for first:
Failed 2 bad-77dc8dbb75-krx5k FailedScheduling 1 toobigTwo problems in two lines, out of the twenty events in that namespace.
The middle column is the count, and it is worth understanding because it explains why the numbers never quite add up. Kubernetes deduplicates repeated events: the same reason, on the same object, from the same source is not written again, its count is incremented and
lastTimestampupdated. So a Pod that has retried an image pull forty times has one event withcount: 40, not forty events.That is why
describeshows ages like23s (x2 over 45s). Two occurrences, the most recent 23 seconds ago, the first 45 seconds ago.The count is diagnostic in its own right. Compare
firstTimestampandlastTimestamp:Pulling 2 2026-08-21T11:59:11Z 2026-08-21T11:59:33Z Failed 2 2026-08-21T11:59:17Z 2026-08-21T11:59:52ZA high count over a long window is an ongoing failure. A high count over a few seconds is a tight crash loop. A count of 1 from ten minutes ago is a transient that has already resolved, and chasing it is usually a waste of time. That distinction, live problem against historical noise, is not visible without the timestamps.
One thing the count hides: because only the first occurrence keeps its message, a failure whose message *changes* between occurrences shows you the old text. Rare, but it explains the occasional event whose message does not match what the Pod is currently doing.
bash Example session kubectl get events -n tri --field-selector type=Warning -o custom-columns=REASON:.reason,COUNT:.count,OBJECT:.involvedObject.name --no-headers | sort -u | head -6Failed 2 bad-77dc8dbb75-krx5kFailedScheduling 1 toobigkubectl get events -n tri -o custom-columns=REASON:.reason,COUNT:.count,FIRST:.firstTimestamp,LAST:.lastTimestamp --no-headers | sort -k2 -rn | head -4Pulling 2 2026-08-21T11:59:11Z 2026-08-21T11:59:33ZFailed 2 2026-08-21T11:59:17Z 2026-08-21T11:59:36ZFailed 2 2026-08-21T11:59:17Z 2026-08-21T11:59:52ZFailed 2 2026-08-21T11:59:17Z 2026-08-21T11:59:36ZExpected resultTwo warnings from twenty events, and counts above 1 with a spread between first and last timestamp. The three
Failedrows are separate events,ErrImagePull,ImagePullBackOffand the pull error, which is why the reason column repeats.Success conditionFiltering to warnings reduces the namespace to its actual problems, and the count column reads as repetition rather than distinct failures.
-
Events expire, and this catches everyone
Twenty events in the namespace, five warnings across the whole cluster:
20 5Those numbers are small for a reason. Events are stored in etcd with a TTL and deleted when it expires. The default is one hour, set by
--event-ttlon the API server, and this install does not override it:no --event-ttl flag set, default 1h appliesSo an incident from this morning has no events left. "I checked and there were no events" and "nothing was wrong" are different statements, and confusing them sends people looking in the wrong place. When the events for a failure are gone, the object's
statusis what remains, which is step 5.Two more properties to keep in mind.
Events are namespaced.
kubectl get eventswith no-nshows only the current namespace. Cluster-scoped failures, and anything in a namespace you were not thinking about, need-A. The five-warning cluster-wide count above included aFailedMountiningress-nginxthat the namespace view could not see.Events are not an audit log. They are a best-effort notification stream, rate-limited per source and dropped under pressure. Anything that needs to survive an hour, or needs to be complete, belongs in the audit log or in a monitoring system that scrapes events and stores them. Several tools do exactly that, and in a cluster you operate seriously, having events shipped somewhere durable is worth the setup.
bash Example session kubectl get events -n tri --no-headers 2>/dev/null | wc -l20kubectl get events -A --field-selector type=Warning --no-headers 2>/dev/null | wc -l5sudo grep -o 'event-ttl=[^ ]*' /etc/kubernetes/manifests/kube-apiserver.yaml || echo 'no --event-ttl flag set, default 1h applies'no --event-ttl flag set, default 1h appliesExpected resultA small number of events, and no
--event-ttlin the API server manifest, so the one-hour default applies. The||branch is what printed, which is the point: the flag is absent.Success conditionYou know how long events survive on this cluster and that the default is one hour.
-
describe: the events for one object, already filtered
kubectl describeends with the events for that object, which is usually faster than filtering the event list yourself:Events: Type Reason Age From Message Normal Scheduled 46s default-scheduler Successfully assigned tri/bad-77dc8dbb75-krx5k to cka1001-node02 Normal Pulling 23s (x2 over 45s) kubelet spec.containers{nginx}: Pulling image "nginx:does-not-exist-1234" Warning Failed 20s (x2 over 39s) kubelet spec.containers{nginx}: Failed to pull image ... not found Warning Failed 20s (x2 over 39s) kubelet spec.containers{nginx}: Error: ErrImagePull Normal BackOff 4s (x2 over 39s) kubelet spec.containers{nginx}: Back-off pulling image "nginx:does-not-exist-1234"The From column is the one people skip, and it is the most orienting field in the block. It names the component that wrote the event, and that tells you which half of the system to investigate:
default-scheduler- placement. A Pod with only scheduler events has never been assigned to a node.kubelet- everything after placement: pulls, mounts, probes, starts, OOM kills.deployment-controller,replicaset-controller- the controllers reconciling higher-level objects.nginx-ingress-controller,calico, and similar - whichever add-on is involved.
A
Scheduledevent fromdefault-schedulerfollowed by kubelet failures, as above, is a clear story: the Pod was placed successfully and the node could not run it. NoScheduledevent at all means the scheduler never placed it, and nothing about the container matters yet.spec.containers{nginx}in each message is the field path, which identifies the specific container in a multi-container Pod. On a Pod with an init container and three sidecars, that prefix is how you tell which one is failing.On a Deployment, remember step 1: describing the Deployment shows Deployment events, mostly scaling. The Pod-level failures are on the Pods, and getting there means
kubectl describe podon one of them.bash Example session kubectl describe pod bad-77dc8dbb75-krx5k -n tri | sed -n '/^Events:/,$p' | head -8Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 46s default-scheduler Successfully assigned tri/bad-77dc8dbb75-krx5k to cka1001-node02 Normal Pulling 23s (x2 over 45s) kubelet spec.containers{nginx}: Pulling image "nginx:does-not-exist-1234" Warning Failed 20s (x2 over 39s) kubelet spec.containers{nginx}: Failed to pull image "nginx:does-not-exist-1234": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:does-not-exist-1234": failed to resolve image: docker.io/library/nginx:does-not-exist-1234: not found Warning Failed 20s (x2 over 39s) kubelet spec.containers{nginx}: Error: ErrImagePull Normal BackOff 4s (x2 over 39s) kubelet spec.containers{nginx}: Back-off pulling image "nginx:does-not-exist-1234"Expected resultThe events for that Pod only, with a From column and
(xN over M)repetition counts. Thesedis only trimming the long describe output down to the Events block.Success conditionYou can read the Events block from describe and tell scheduler problems from kubelet problems by the From column.
-
Status is the durable record
Events expire. Status does not. It lives on the object and is updated continuously, which makes it the right thing to read for a current state and the only thing left once the events are gone.
Three places to look, in increasing usefulness.
Container status carries the specific reason:
nginx ready=false reason=ErrImagePullThat single line replaces reading the whole event stream. The
stateof a container is one ofwaiting,running, orterminated, and each carries its own detail:state.waiting.reasonhere,state.terminated.exitCodeandreasonfor something that ran and stopped.lastState.terminatedholds the *previous* run, which is where the exit code of the crash before the current restart lives, and that is exactly what you need for a crash loop.Phase against readiness is the pair worth reading together:
Pending false ErrImagePullA Pod is
Pendinguntil every container has started, so an image that cannot be pulled leaves it Pending indefinitely even though it was scheduled successfully. Phase is coarse,Pending,Running,Succeeded,Failed,Unknown, and a Pod that isRunningwhile notReadyis completely normal, which is whykubectl get podsshows the READY column separately.Conditions are the structured form, and the one to build tooling on:
PodReadyToStartContainers=True Initialized=True Ready=False ContainersReady=False PodScheduled=TrueRead left to right and it is a checklist of how far the Pod got.
PodScheduled=TrueandReady=Falsemeans placement succeeded and the containers did not come up, and you did not need an event to know it.Higher-level objects have their own conditions, and the
reasonis the payload:bad: Available=False (MinimumReplicasUnavailable) Progressing=True (ReplicaSetUpdated) good: Available=True (MinimumReplicasAvailable) Progressing=True (NewReplicaSetAvailable)MinimumReplicasUnavailableagainstMinimumReplicasAvailableis a one-word diagnosis. Note that the broken Deployment is stillProgressing=True: it has not given up. AfterprogressDeadlineSeconds, ten minutes by default, that flips toProgressing=Falsewith reasonProgressDeadlineExceeded, which is the signal a deployment has genuinely stalled rather than merely being slow. A CI job that waits without checking for that condition waits forever.Conditions are also what
kubectl waitreads, which makes them scriptable:deployment.apps/good condition met error: timed out waiting for the condition on deployments/badA non-zero exit on timeout, so
kubectl wait --for=condition=Available deploy/x --timeout=60sis a usable gate in a pipeline.kubectl wait --for=jsonpath='{.status.phase}'=Succeededcovers the cases where no condition says what you need.bash Example session kubectl get pod bad-77dc8dbb75-krx5k -n tri -o jsonpath='{range .status.containerStatuses[*]}{.name}{" ready="}{.ready}{" reason="}{.state.waiting.reason}{"\n"}{end}'nginx ready=false reason=ErrImagePullkubectl get pod bad-77dc8dbb75-krx5k -n tri -o custom-columns=PHASE:.status.phase,READY:.status.containerStatuses[0].ready,REASON:.status.containerStatuses[0].state.waiting.reason --no-headersPending false ErrImagePullkubectl get pod bad-77dc8dbb75-krx5k -n tri -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{end}{"\n"}'PodReadyToStartContainers=True Initialized=True Ready=False ContainersReady=False PodScheduled=True kubectl get deploy bad -n tri -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" ("}{.reason}{")\n"}{end}'Available=False (MinimumReplicasUnavailable)Progressing=True (ReplicaSetUpdated)kubectl get deploy good -n tri -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" ("}{.reason}{")\n"}{end}'Available=True (MinimumReplicasAvailable)Progressing=True (NewReplicaSetAvailable)kubectl wait --for=condition=Available deploy/good -n tri --timeout=60sdeployment.apps/good condition metkubectl wait --for=condition=Available deploy/bad -n tri --timeout=10serror: timed out waiting for the condition on deployments/badExpected resultThe failure reason from status rather than from an event, and matching pairs of conditions on a working and a broken Deployment. The
waitonbadreturns a non-zero exit, which is what makes it usable in a script.Success conditionYou can read a failure reason out of
.statusand use conditions withkubectl wait.
Troubleshooting
A Deployment is broken and
kubectl describe deploymentshows no useful events.Why: Events attach to the object a component acted on. Pull, mount and probe failures happen to Pods, so they are recorded against Pods.
Fix:Go down a level:
kubectl get pods -l <selector>thenkubectl describe pod <name>. Or filter the namespace:kubectl get events -n <ns> --field-selector type=Warning. The Deployment's own events are about scaling and rollouts, which is a different question.There are no events for a failure you know happened.
Why: Events expire, one hour by default, and they are best-effort so they can be dropped under load.
Fix:Read
.statusinstead, which persists:state.waiting.reasonfor something not started,lastState.terminated.exitCodefor the run before the current restart, and the object's conditions.--event-ttlon the API server raises the retention, and shipping events to a monitoring system is the real answer for anything you need later.The event list is out of order and hard to follow.
Why: Events are not returned chronologically by default.
Fix:
kubectl get events --sort-by=.lastTimestamp. Worth typing every time. Add-Afor the whole cluster and--field-selector type=Warningto cut the routine chatter.An event's count is high but you cannot tell if the problem is current.
Why: Deduplication means one event covers many occurrences, so the count says nothing about when.
Fix:Compare
firstTimestampandlastTimestamp. A recentlastTimestampmeans it is still happening; an old one with a high count is a resolved incident.describeshows the same thing as23s (x2 over 45s).A Pod is
Runningbut the application is not working.Why: Phase and readiness are different things.
Runningmeans the containers started, not that they are serving.Fix:Read the READY column and
.status.containerStatuses[].ready. A Running Pod that is not Ready has a failing readiness probe, andkubectl describe podshows the probe failures. A Pod that is Ready with a broken application is an application problem, so go to the container logs.A CI job waiting for a rollout never finishes.
Why: It is waiting on
Availableand nothing ever sets it, and the job has no timeout of its own.Fix:Give
kubectl waita--timeoutand check the exit code, or watch forProgressing=Falsewith reasonProgressDeadlineExceeded, which is Kubernetes' own "this rollout has stalled" signal afterprogressDeadlineSeconds.kubectl rollout status --timeout=does both.Events are missing in one namespace but present in another.
Why: Events are namespaced, and
kubectl get eventsdefaults to the current namespace.Fix:Use
-A. A cluster-wide warning sweep,kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp, is the first command worth running on a cluster you do not know.