Hands-on Lab·Kubernetes and Cloud Native Associate

Resource Requests, Usage and Cost

A cluster can be completely full while every node sits at 3% CPU. The scheduler spends what you reserve, not what you use, which is why cost conversations in Kubernetes are really conversations about requests.

Cloud Native Architecture Guide 34 of 46 Beginner

Kubernetes1.36.4
metrics-serverinstalled (required for kubectl top)
TimeAbout 15 min
Reviewed22 August 2026

Written against the versions above. Requests, allocatable and the scheduler's fit check are long-stable. `kubectl top` needs metrics-server, which this cluster already runs.

A four-node kubeadm cluster. Every node is 2 vCPU, which is what makes the reservation arithmetic in this guide small enough to follow.
Server Name	IP Address	OS	Roles	CPU	RAM	HDD
CKA1001	192.168.0.175	Ubuntu 26.04 LTS	Control Plane Node	2 Core	4 GB	50 GB
CKA1001-NODE01	192.168.0.176	Ubuntu 26.04 LTS	Worker Node	2 Core	4 GB	50 GB
CKA1001-NODE02	192.168.0.177	Ubuntu 26.04 LTS	Worker Node	2 Core	4 GB	50 GB
CKA1001-NODE03	192.168.0.178	Ubuntu 26.04 LTS	Worker Node	2 Core	4 GB	50 GB

Before you start

A cluster with metrics-server, so kubectl top returns real numbers.
Permission to create a namespace and scale a Deployment in it.

What a node actually sells
A node reports two numbers, and only one of them is for sale.

capacity is the hardware. allocatable is what is left for your Pods after the kubelet reserves memory and CPU for itself, the container runtime and the OS. Allocatable is the number the scheduler works from, and it is always smaller than capacity.

On a 2 vCPU node the gap is small in CPU and noticeable in memory. Reading capacity and planning against it is a common way to end up with Pods that will not fit on a node you were sure had room.
bash Example session
```
$kubectl --context cka1001 get nodes -o custom-columns=NAME:.metadata.name,CPU_CAP:.status.capacity.cpu,CPU_ALLOC:.status.allocatable.cpu,MEM_CAP:.status.capacity.memory,MEM_ALLOC:.status.allocatable.memoryNAME             CPU_CAP   CPU_ALLOC   MEM_CAP     MEM_ALLOCcka1001          2         2           3480372Ki   3377972Kicka1001-node01   2         2           3480380Ki   3377980Kicka1001-node02   2         2           3480380Ki   3377980Kicka1001-node03   2         2           3480380Ki   3377980Ki
```
Expected resultTwo CPU numbers and two memory numbers per node, with allocatable below capacity.

Success conditionYou can see the difference between what the node has and what it will let you book.

Reserved is not used

Two views of the same node, five seconds apart.

kubectl top reads live consumption from metrics-server. describe node reports Allocated resources, which is the sum of requests of everything scheduled there. They answer different questions, and the second one is the one that costs money.

Note the last pair of numbers. Of the Pods on this cluster, most declare no CPU request at all - they consume real CPU while booking none of it, which is why a cluster can look busy in top and empty to the scheduler at the same time.

bash Example session

$kubectl --context cka1001 top nodesNAME             CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)cka1001          116m         5%       2310Mi          70%cka1001-node01   103m         5%       1613Mi          48%cka1001-node02   77m          3%       1906Mi          57%cka1001-node03   56m          2%       1441Mi          43%$kubectl describe node cka1001-node01 | sed -n '/Allocated resources/,/^Events/p' | head -12Allocated resources:  (Total limits may be over 100 percent, i.e., overcommitted.)  Resource           Requests     Limits  --------           --------     ------  cpu                135m (6%)    200m (10%)  memory             518Mi (15%)  298Mi (9%)  ephemeral-storage  0 (0%)       0 (0%)  hugepages-1Gi      0 (0%)       0 (0%)  hugepages-2Mi      0 (0%)       0 (0%)Events:              <none>$kubectl --context cka1001 get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,CPUREQ:.spec.containers[0].resources.requests.cpu --no-headers | awk '$3=="<none>"' | wc -l69$kubectl --context cka1001 get pods -A --no-headers | wc -l91

Expected resultNodes at single-digit CPU percentages, an Allocated resources block in the same range, and a count of Pods with no CPU request.

Success conditionYou have two different numbers for the same node and know which one the scheduler reads.

Reserve half a core and use none of it

Two replicas of sleep 3600, each requesting 500m CPU and 256Mi of memory.

sleep uses no CPU. That is the point: the request is a claim on the node, honoured whether or not anything is consumed. The scheduler subtracts 500m from allocatable the moment the Pod is placed, and does not give it back when the process turns out to be idle.

Watch node01's reservation move from 135m to 635m while its actual usage does not change at all.

bash Example session

$kubectl --context cka1001 apply -f - <<'YAML'apiVersion: apps/v1kind: Deploymentmetadata:  name: idle-hog  namespace: cost-demospec:  replicas: 2  selector:    matchLabels: { app: idle-hog }  template:    metadata:      labels: { app: idle-hog }    spec:      containers:      - name: sleeper        image: busybox:1.37        command: ["sleep", "3600"]        resources:          requests:            cpu: "500m"            memory: "256Mi"YAML$kubectl --context cka1001 -n cost-demo top podsNAME                        CPU(cores)   MEMORY(bytes)idle-hog-57d5fc9545-qfnz8   0m           0Miidle-hog-57d5fc9545-tgnpb   0m           0Mi$kubectl --context cka1001 -n cost-demo get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,CPUREQ:.spec.containers[0].resources.requests.cpu --no-headersidle-hog-57d5fc9545-qfnz8   cka1001-node01   500midle-hog-57d5fc9545-tgnpb   cka1001-node03   500m$kubectl --context cka1001 describe node cka1001-node01 | grep -E 'cpu +[0-9]+m? +\(' | head -3  cpu                635m (31%)   200m (10%)

Expected resultBoth Pods reporting 0m of CPU, each holding a 500m reservation, and the node's allocated total up by a full core.

Success conditionThe reservation moved and the usage did not.

Full at zero percent
Now scale to twelve and watch the cluster refuse.

Nine replicas schedule. Three stay Pending, and the scheduler says why: Insufficient cpu on three nodes, with the fourth ruled out by the control plane's taint. Meanwhile the total CPU actually being used by all twelve replicas is 0m.

This is the whole lesson. The cluster is out of capacity at close to zero utilisation, because capacity is spent by reservation. On a cloud provider that is the moment you add a node and start paying for it - not because the workload needs the compute, but because somebody typed a number into requests.

It also decides who owns the cost. The number lives in the application's own manifest, so the developer writing that manifest sets the bill, and the platform team that owns the node budget cannot see it until the Pods land. Right-sizing requests is the highest-leverage cost work in Kubernetes for exactly that reason.
bash Example session
```
$kubectl --context cka1001 -n cost-demo scale deploy idle-hog --replicas=12deployment.apps/idle-hog scaled$kubectl --context cka1001 -n cost-demo get pods --no-headers | awk '{print $3}' | sort | uniq -c      3 Pending      9 Running$kubectl --context cka1001 -n cost-demo get pods --field-selector=status.phase=Pending -o custom-columns=POD:.metadata.name,REASON:.status.conditions[0].reason --no-headers | head -3idle-hog-57d5fc9545-tm2pq   Unschedulableidle-hog-57d5fc9545-trxlm   Unschedulableidle-hog-57d5fc9545-zs8dc   Unschedulable$kubectl --context cka1001 -n cost-demo describe pod -l app=idle-hog | grep -m1 -A2 'Insufficient cpu' || echo "no Insufficient cpu message"  Warning  FailedScheduling  39s   default-scheduler  0/4 nodes are available: 1 node(s) had untolerated taint(s), 3 Insufficient cpu. no new claims to deallocate, preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.$kubectl --context cka1001 -n cost-demo top pods --no-headers | awk '{s+=$2} END {print "actual cpu used by all replicas: " s "m"}'actual cpu used by all replicas: 0m
```
Expected resultThree Pending Pods, an Insufficient cpu scheduling message, and 0m of CPU in use.

Success conditionYou have a full cluster and an idle one at the same time, and you know why both are true.

Troubleshooting

kubectl top returns error: Metrics API not available.

Why: metrics-server is not installed, or its Pod is not ready.

Fix:kubectl -n kube-system get deploy metrics-server. Without it you can still read requests from describe node; only the live-usage half of this guide needs it.
Pods stay Pending and describe says Insufficient cpu, but top nodes shows the node nearly idle.

Why: This is not a fault. The scheduler fits Pods against the sum of requests, not against live usage.

Fix:Lower the requests to what the workload actually needs, or add capacity. Requests are a booking, not a limit.
A namespace with no obvious workloads still reports a large allocated total.

Why: DaemonSets. They place a Pod on every node and their requests count on every node.

Fix:kubectl get ds -A. On small nodes the agents can be a significant share of allocatable before you deploy anything.

Official sources

← All KCNA guides