CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Administrator

Metrics Server and Resource Pressure

The same node reads 1% CPU in kubectl top and 5% in describe node. Both are right, they measure different things, and confusing them is why clusters report Insufficient cpu while every dashboard shows idle machines.

Troubleshooting Guide 79 of 103 Intermediate

Written against the versions above. metrics-server is not installed by default. Without it, kubectl top returns an error rather than zeros.

metrics-server runs as a Deployment in kube-system.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. What kubectl top measures

    kubectl top reports actual consumption, sampled. metrics-server scrapes each kubelet's summary API every 15 seconds, keeps only the latest value in memory, and serves it through the Metrics API.

    Three properties follow from that design and each one matters:

    • There is no history. metrics-server holds one sample. kubectl top cannot tell you what happened an hour ago, and there is no flag to make it. For history you need Prometheus or equivalent.
    • It is a sample, not a peak. A container that spikes to its memory limit between scrapes is killed without ever appearing in top, which is why the OOMKilled guide warns against trusting these numbers for limit-setting.
    • It is in memory only. Restart the Pod and the data starts again from nothing, so kubectl top returns errors for a minute or two after any metrics-server restart.

    Reading the node output: CPU in millicores against the node's capacity, memory in bytes. cka1001 at 98m and 41% memory is the control plane carrying the API server and etcd; the workers are near idle.

    The Pod view sorted by memory shows where the control plane's memory actually goes: kube-apiserver at 484Mi, six times etcd's 79Mi. That ratio is normal and worth knowing, because the API server's memory scales with the number of objects and watchers rather than with your workload.

    --containers breaks a Pod down per container, which is what you need when a Pod has a sidecar and you are trying to work out which half is consuming.

    One caveat on the small numbers: sidecar 0Mi does not mean zero. kubectl top rounds to whole mebibytes, so anything under about half a mebibyte displays as 0.

    bash Example session 
    kubectl top nodesNAME             CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   cka1001          98m          4%       1385Mi          41%         cka1001-node01   29m          1%       972Mi           29%         cka1001-node02   37m          1%       857Mi           26%         cka1001-node03   29m          1%       598Mi           18%         kubectl top pods -n kube-system --sort-by=memory | head -6NAME                              CPU(cores)   MEMORY(bytes)   kube-apiserver-cka1001            26m          484Mi           etcd-cka1001                      14m          79Mi            coredns-589f44dc88-fdcml          1m           68Mi            kube-controller-manager-cka1001   7m           62Mi            kube-scheduler-cka1001            3m           26Mi            kubectl top pod app -n dbg --containersPOD   NAME      CPU(cores)   MEMORY(bytes)   app   sidecar   1m           0Mi             app   web       1m           3Mi

    Expected resultLive usage for nodes, Pods and containers. --sort-by=memory is the flag that makes the Pod view useful; unsorted output on a busy namespace is unreadable.

    Success conditionkubectl top nodes returns numbers rather than an error.

  2. The same node, a different 1%

    This is the point of the guide. kubectl top said cka1001-node01 was using 1% CPU and 29% memory. describe node says:

    cpu                100m (5%)  0 (0%)
memory             70Mi (2%)  170Mi (5%)

    5% CPU and 2% memory. Different numbers for the same node at the same moment, and both correct, because they measure different things:

    • kubectl top - what is being used right now.
    • describe node - what has been requested and limited, summed across the node's Pods.

    The scheduler uses the second one. Exclusively. It has no idea what anything is actually consuming; it packs Pods onto nodes by comparing requests against allocatable.

    That resolves the contradiction that puzzles people most often: Insufficient cpu on a cluster where every node looks idle. A node whose Pods request all its CPU is full, at any level of real utilisation. Adding nodes fixes it; adding load does not cause it.

    So when a Pod will not schedule, kubectl top is the wrong command. kubectl describe node and its Allocated resources table is the right one, and the percentages there are of allocatable, not capacity, which is why they can approach 100% while the machine has memory free.

    The header on that table is worth reading too: (Total limits may be over 100 percent, i.e., overcommitted.). Limits are allowed to exceed the node, because they are ceilings rather than reservations. Requests are not: their sum cannot exceed allocatable, and that is the constraint the scheduler enforces.

    On this node, requests at 5% and usage at 1% is a modest gap. In production the gap is usually much larger and always in that direction, because people set requests from a guess and the guess is high. That over-request is the most common cause of a cluster that costs too much while looking empty.

    bash Example session 
    kubectl describe node cka1001-node01 | sed -n '/Allocated resources/,/Events/p' | head -12Allocated resources:  (Total limits may be over 100 percent, i.e., overcommitted.)  Resource           Requests   Limits  --------           --------   ------  cpu                100m (5%)  0 (0%)  memory             70Mi (2%)  170Mi (5%)  ephemeral-storage  0 (0%)     0 (0%)  hugepages-1Gi      0 (0%)     0 (0%)  hugepages-2Mi      0 (0%)     0 (0%)Events:              <none>

    Expected resultRequests and limits side by side. cpu ... 0 (0%) in the Limits column means no Pod on this node sets a CPU limit, which is common and usually correct: a CPU limit throttles rather than kills.

    Success conditionYou can state the node's requested CPU and its used CPU as two different numbers.

  3. The conditions that mean pressure

    Usage and requests are both about capacity planning. The node's conditions are about whether the kubelet is currently in trouble, and they are what triggers eviction.

    NetworkUnavailable=False MemoryPressure=False DiskPressure=False PIDPressure=False Ready=True

    Five conditions, and the polarity is mixed, which is a genuine source of misreading: for the four pressure conditions False is healthy, and for Ready it is True.

    What each one means when it flips:

    • MemoryPressure=True - available memory has fallen below the eviction threshold. The kubelet starts evicting Pods, choosing by QoS class: BestEffort first, then Burstable exceeding its requests, and Guaranteed last. It also adds a taint that stops new Pods arriving.
    • DiskPressure=True - the image or root filesystem is low. The kubelet first garbage-collects unused images, which is often enough, then evicts.
    • PIDPressure=True - the node is near its process limit. Rare, and usually a runaway fork loop.
    • Ready=False - the kubelet has stopped reporting or reports itself unhealthy. Covered in its own guide.

    The important distinction from the previous step: eviction responds to real usage, scheduling responds to requests. A node can be perfectly schedulable and evicting Pods, or under memory pressure while its requests look modest. They are separate mechanisms and the same node can be in trouble on one and fine on the other.

    And a note on where the eviction thresholds come from: they are the kubelet's, not the scheduler's, and they are set on the node (--eviction-hard, default around 100Mi of available memory). That is why allocatable is smaller than capacity: the difference is what the kubelet reserves for the system and for its eviction headroom.

    bash Example session 
    kubectl get node cka1001-node01 -o jsonpath="{range .status.conditions[*]}{.type}={.status}{\" \"}{end}{\"\n\"}"NetworkUnavailable=False MemoryPressure=False DiskPressure=False PIDPressure=False Ready=True

    Expected resultAll four pressure conditions False and Ready True, which is a healthy node. Scan this rather than kubectl top when Pods are being evicted.

    Success conditionYou can name which polarity is healthy for each condition.

Troubleshooting

Official sources

← All CKA guides