CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Administrator

Events, describe and Object Status

Almost every Kubernetes failure explains itself in one of three places: an event, the Events section of describe, or a status condition. Knowing which of the three to read, and that events expire after an hour, is most of the skill.

Troubleshooting Guide 76 of 103 Beginner

Written against the versions above. Event reasons and condition names are stable across recent versions. The exact wording of messages is not, so read them for meaning rather than matching strings.

Two deliberately broken workloads on the four-node cluster: one with an image that does not exist, one asking for more CPU and memory than any node has.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. What an event actually is

    An event is a short record written by a component about an object: the scheduler saying it could not place a Pod, the kubelet saying an image pull failed. Two broken workloads produce a readable trail:

    Warning   FailedScheduling  toobig  0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory...
    Warning   Failed  bad-77dc8dbb75-krx5k  Failed to pull image "nginx:does-not-exist-1234": rpc error: code = NotFound...
    Warning   Failed  bad-77dc8dbb75-krx5k  Error: ErrImagePull
    Normal    BackOff bad-77dc8dbb75-krx5k  Back-off pulling image "nginx:does-not-exist-1234"

    Four fields carry the information.

    Type is Normal or Warning, and nothing else. That makes --field-selector type=Warning the single highest-value filter in Kubernetes troubleshooting: it turns a wall of routine pull-and-start chatter into the short list of things that went wrong.

    Reason is a short CamelCase token from a fixed vocabulary: FailedScheduling, ErrImagePull, BackOff, Unhealthy, OOMKilling, FailedMount. Reasons are what you filter and search on, because they are stable across versions in a way messages are not.

    involvedObject names what the event is about, and this is the field that trips people up. Events attach to the object a component was acting on, which is almost always the Pod, not the Deployment. Asking for events on a Deployment that is failing because its Pods cannot pull an image returns almost nothing useful, and the conclusion "there are no events" is wrong: they are on the ReplicaSet's Pods.

    Message is free text, and the FailedScheduling one above is a good example of how much it can carry. 0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory accounts for every node in the cluster and says exactly why each was rejected. Scheduler messages of that shape are worth reading in full rather than skimming.

    Note --sort-by=.lastTimestamp on the command. Default event ordering is not chronological, which is confusing enough that the sort flag is worth typing every time.

    bash Example session
    kubectl get events -n tri --sort-by=.lastTimestamp -o custom-columns=TYPE:.type,REASON:.reason,OBJECT:.involvedObject.name,MESSAGE:.message --no-headers | tail -8Normal    Pulled              good-79f4495545-wbc6x   Container image "nginx:1.29-alpine" already present on machine and can be accessed by the podNormal    Created             good-79f4495545-wbc6x   Container createdWarning   FailedScheduling    toobig                  0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu, 3 Insufficient memory. no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.Normal    Pulling             bad-77dc8dbb75-krx5k    Pulling image "nginx:does-not-exist-1234"Warning   Failed              bad-77dc8dbb75-krx5k    Error: ErrImagePullWarning   Failed              bad-77dc8dbb75-krx5k    Failed to pull image "nginx:does-not-exist-1234": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:does-not-exist-1234": failed to resolve image: docker.io/library/nginx:does-not-exist-1234: not foundWarning   Failed              bad-77dc8dbb75-krx5k    Error: ImagePullBackOffNormal    BackOff             bad-77dc8dbb75-krx5k    Back-off pulling image "nginx:does-not-exist-1234"

    Expected resultNormal events for the healthy Deployment mixed with warnings for the two broken workloads. The good Pod's Pulled event says "already present on machine", which is what a cached image looks like.

    Success conditionYou can list events with type, reason, object and message as separate columns.

  2. Filter to warnings, and read the count

    The filtered view is the one to reach for first:

    Failed             2     bad-77dc8dbb75-krx5k
    FailedScheduling   1     toobig

    Two problems in two lines, out of the twenty events in that namespace.

    The middle column is the count, and it is worth understanding because it explains why the numbers never quite add up. Kubernetes deduplicates repeated events: the same reason, on the same object, from the same source is not written again, its count is incremented and lastTimestamp updated. So a Pod that has retried an image pull forty times has one event with count: 40, not forty events.

    That is why describe shows ages like 23s (x2 over 45s). Two occurrences, the most recent 23 seconds ago, the first 45 seconds ago.

    The count is diagnostic in its own right. Compare firstTimestamp and lastTimestamp:

    Pulling   2   2026-08-21T11:59:11Z   2026-08-21T11:59:33Z
    Failed    2   2026-08-21T11:59:17Z   2026-08-21T11:59:52Z

    A high count over a long window is an ongoing failure. A high count over a few seconds is a tight crash loop. A count of 1 from ten minutes ago is a transient that has already resolved, and chasing it is usually a waste of time. That distinction, live problem against historical noise, is not visible without the timestamps.

    One thing the count hides: because only the first occurrence keeps its message, a failure whose message *changes* between occurrences shows you the old text. Rare, but it explains the occasional event whose message does not match what the Pod is currently doing.

    bash Example session
    kubectl get events -n tri --field-selector type=Warning -o custom-columns=REASON:.reason,COUNT:.count,OBJECT:.involvedObject.name --no-headers | sort -u | head -6Failed             2     bad-77dc8dbb75-krx5kFailedScheduling   1     toobigkubectl get events -n tri -o custom-columns=REASON:.reason,COUNT:.count,FIRST:.firstTimestamp,LAST:.lastTimestamp --no-headers | sort -k2 -rn | head -4Pulling             2     2026-08-21T11:59:11Z   2026-08-21T11:59:33ZFailed              2     2026-08-21T11:59:17Z   2026-08-21T11:59:36ZFailed              2     2026-08-21T11:59:17Z   2026-08-21T11:59:52ZFailed              2     2026-08-21T11:59:17Z   2026-08-21T11:59:36Z

    Expected resultTwo warnings from twenty events, and counts above 1 with a spread between first and last timestamp. The three Failed rows are separate events, ErrImagePull, ImagePullBackOff and the pull error, which is why the reason column repeats.

    Success conditionFiltering to warnings reduces the namespace to its actual problems, and the count column reads as repetition rather than distinct failures.

  3. Events expire, and this catches everyone

    Twenty events in the namespace, five warnings across the whole cluster:

    20
    5

    Those numbers are small for a reason. Events are stored in etcd with a TTL and deleted when it expires. The default is one hour, set by --event-ttl on the API server, and this install does not override it:

    no --event-ttl flag set, default 1h applies

    So an incident from this morning has no events left. "I checked and there were no events" and "nothing was wrong" are different statements, and confusing them sends people looking in the wrong place. When the events for a failure are gone, the object's status is what remains, which is step 5.

    Two more properties to keep in mind.

    Events are namespaced. kubectl get events with no -n shows only the current namespace. Cluster-scoped failures, and anything in a namespace you were not thinking about, need -A. The five-warning cluster-wide count above included a FailedMount in ingress-nginx that the namespace view could not see.

    Events are not an audit log. They are a best-effort notification stream, rate-limited per source and dropped under pressure. Anything that needs to survive an hour, or needs to be complete, belongs in the audit log or in a monitoring system that scrapes events and stores them. Several tools do exactly that, and in a cluster you operate seriously, having events shipped somewhere durable is worth the setup.

    bash Example session
    kubectl get events -n tri --no-headers 2>/dev/null | wc -l20kubectl get events -A --field-selector type=Warning --no-headers 2>/dev/null | wc -l5sudo grep -o 'event-ttl=[^ ]*' /etc/kubernetes/manifests/kube-apiserver.yaml || echo 'no --event-ttl flag set, default 1h applies'no --event-ttl flag set, default 1h applies

    Expected resultA small number of events, and no --event-ttl in the API server manifest, so the one-hour default applies. The || branch is what printed, which is the point: the flag is absent.

    Success conditionYou know how long events survive on this cluster and that the default is one hour.

  4. describe: the events for one object, already filtered

    kubectl describe ends with the events for that object, which is usually faster than filtering the event list yourself:

    Events:
      Type     Reason     Age                From               Message
      Normal   Scheduled  46s                default-scheduler  Successfully assigned tri/bad-77dc8dbb75-krx5k to cka1001-node02
      Normal   Pulling    23s (x2 over 45s)  kubelet            spec.containers{nginx}: Pulling image "nginx:does-not-exist-1234"
      Warning  Failed     20s (x2 over 39s)  kubelet            spec.containers{nginx}: Failed to pull image ... not found
      Warning  Failed     20s (x2 over 39s)  kubelet            spec.containers{nginx}: Error: ErrImagePull
      Normal   BackOff    4s   (x2 over 39s)  kubelet            spec.containers{nginx}: Back-off pulling image "nginx:does-not-exist-1234"

    The From column is the one people skip, and it is the most orienting field in the block. It names the component that wrote the event, and that tells you which half of the system to investigate:

    • default-scheduler - placement. A Pod with only scheduler events has never been assigned to a node.
    • kubelet - everything after placement: pulls, mounts, probes, starts, OOM kills.
    • deployment-controller, replicaset-controller - the controllers reconciling higher-level objects.
    • nginx-ingress-controller, calico, and similar - whichever add-on is involved.

    A Scheduled event from default-scheduler followed by kubelet failures, as above, is a clear story: the Pod was placed successfully and the node could not run it. No Scheduled event at all means the scheduler never placed it, and nothing about the container matters yet.

    spec.containers{nginx} in each message is the field path, which identifies the specific container in a multi-container Pod. On a Pod with an init container and three sidecars, that prefix is how you tell which one is failing.

    On a Deployment, remember step 1: describing the Deployment shows Deployment events, mostly scaling. The Pod-level failures are on the Pods, and getting there means kubectl describe pod on one of them.

    bash Example session
    kubectl describe pod bad-77dc8dbb75-krx5k -n tri | sed -n '/^Events:/,$p' | head -8Events:  Type     Reason     Age                From               Message  ----     ------     ----               ----               -------  Normal   Scheduled  46s                default-scheduler  Successfully assigned tri/bad-77dc8dbb75-krx5k to cka1001-node02  Normal   Pulling    23s (x2 over 45s)  kubelet            spec.containers{nginx}: Pulling image "nginx:does-not-exist-1234"  Warning  Failed     20s (x2 over 39s)  kubelet            spec.containers{nginx}: Failed to pull image "nginx:does-not-exist-1234": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:does-not-exist-1234": failed to resolve image: docker.io/library/nginx:does-not-exist-1234: not found  Warning  Failed     20s (x2 over 39s)  kubelet            spec.containers{nginx}: Error: ErrImagePull  Normal   BackOff    4s (x2 over 39s)   kubelet            spec.containers{nginx}: Back-off pulling image "nginx:does-not-exist-1234"

    Expected resultThe events for that Pod only, with a From column and (xN over M) repetition counts. The sed is only trimming the long describe output down to the Events block.

    Success conditionYou can read the Events block from describe and tell scheduler problems from kubelet problems by the From column.

  5. Status is the durable record

    Events expire. Status does not. It lives on the object and is updated continuously, which makes it the right thing to read for a current state and the only thing left once the events are gone.

    Three places to look, in increasing usefulness.

    Container status carries the specific reason:

    nginx ready=false reason=ErrImagePull

    That single line replaces reading the whole event stream. The state of a container is one of waiting, running, or terminated, and each carries its own detail: state.waiting.reason here, state.terminated.exitCode and reason for something that ran and stopped. lastState.terminated holds the *previous* run, which is where the exit code of the crash before the current restart lives, and that is exactly what you need for a crash loop.

    Phase against readiness is the pair worth reading together:

    Pending   false   ErrImagePull

    A Pod is Pending until every container has started, so an image that cannot be pulled leaves it Pending indefinitely even though it was scheduled successfully. Phase is coarse, Pending, Running, Succeeded, Failed, Unknown, and a Pod that is Running while not Ready is completely normal, which is why kubectl get pods shows the READY column separately.

    Conditions are the structured form, and the one to build tooling on:

    PodReadyToStartContainers=True Initialized=True Ready=False ContainersReady=False PodScheduled=True

    Read left to right and it is a checklist of how far the Pod got. PodScheduled=True and Ready=False means placement succeeded and the containers did not come up, and you did not need an event to know it.

    Higher-level objects have their own conditions, and the reason is the payload:

    bad:   Available=False (MinimumReplicasUnavailable)   Progressing=True  (ReplicaSetUpdated)
    good:  Available=True  (MinimumReplicasAvailable)     Progressing=True  (NewReplicaSetAvailable)

    MinimumReplicasUnavailable against MinimumReplicasAvailable is a one-word diagnosis. Note that the broken Deployment is still Progressing=True: it has not given up. After progressDeadlineSeconds, ten minutes by default, that flips to Progressing=False with reason ProgressDeadlineExceeded, which is the signal a deployment has genuinely stalled rather than merely being slow. A CI job that waits without checking for that condition waits forever.

    Conditions are also what kubectl wait reads, which makes them scriptable:

    deployment.apps/good condition met
    error: timed out waiting for the condition on deployments/bad

    A non-zero exit on timeout, so kubectl wait --for=condition=Available deploy/x --timeout=60s is a usable gate in a pipeline. kubectl wait --for=jsonpath='{.status.phase}'=Succeeded covers the cases where no condition says what you need.

    bash Example session
    kubectl get pod bad-77dc8dbb75-krx5k -n tri -o jsonpath='{range .status.containerStatuses[*]}{.name}{" ready="}{.ready}{" reason="}{.state.waiting.reason}{"\n"}{end}'nginx ready=false reason=ErrImagePullkubectl get pod bad-77dc8dbb75-krx5k -n tri -o custom-columns=PHASE:.status.phase,READY:.status.containerStatuses[0].ready,REASON:.status.containerStatuses[0].state.waiting.reason --no-headersPending   false   ErrImagePullkubectl get pod bad-77dc8dbb75-krx5k -n tri -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{end}{"\n"}'PodReadyToStartContainers=True Initialized=True Ready=False ContainersReady=False PodScheduled=True kubectl get deploy bad -n tri -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" ("}{.reason}{")\n"}{end}'Available=False (MinimumReplicasUnavailable)Progressing=True (ReplicaSetUpdated)kubectl get deploy good -n tri -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" ("}{.reason}{")\n"}{end}'Available=True (MinimumReplicasAvailable)Progressing=True (NewReplicaSetAvailable)kubectl wait --for=condition=Available deploy/good -n tri --timeout=60sdeployment.apps/good condition metkubectl wait --for=condition=Available deploy/bad -n tri --timeout=10serror: timed out waiting for the condition on deployments/bad

    Expected resultThe failure reason from status rather than from an event, and matching pairs of conditions on a working and a broken Deployment. The wait on bad returns a non-zero exit, which is what makes it usable in a script.

    Success conditionYou can read a failure reason out of .status and use conditions with kubectl wait.

Troubleshooting

Official sources