CertGrid CertGrid
Troubleshooting·Kubernetes and Cloud Native Associate

HorizontalPodAutoscaler and Resource Requests

An HPA on a Deployment with no CPU requests reports cpu: <unknown>/60% forever. A percentage needs a denominator, and requests are the denominator. Same HPA, a Deployment that declares requests, and it reads 2%/60%.

Cloud Native Architecture Guide 30 of 46 Intermediate

Written against the versions above. The interfaces below are versioned standards and change slowly; the implementations plugged into them change constantly. That is the point of them.

A four-node kubeadm cluster with Calico for CNI and two CSI drivers installed.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. The symptom

    Create an HPA against a Deployment and the TARGETS column reads:

    cpu: <unknown>/60%

    Wait for it and it stays that way. REPLICAS sits at the minimum and never moves, whatever the load.

    Note the deprecation warning too: --cpu-percent is on its way out in favour of --cpu=60%. Worth adopting now, since the old flag will eventually stop working.

    bash Example session
    kubectl --context cka1001 -n guestbook autoscale deploy guestbook-ui --cpu-percent=60 --min=1 --max=4horizontalpodautoscaler.autoscaling/guestbook-ui autoscaledFlag --cpu-percent has been deprecated, Use --cpu with percentage or resource quantity format (e.g., '70%' for utilization or '500m' for milliCPU).kubectl --context cka1001 -n guestbook get hpa guestbook-uiNAME           REFERENCE                 TARGETS              MINPODS   MAXPODS   REPLICAS   AGEguestbook-ui   Deployment/guestbook-ui   cpu: <unknown>/60%   1         4         0          3s

    Expected resultcpu: <unknown>/60% rather than a number.

    Success conditionYou have reproduced the unknown target.

  2. The conditions say it plainly

    kubectl get hpa shows the symptom; describe shows the cause. Two conditions matter:

    AbleToScale     True   SucceededGetScale
    ScalingActive   False  FailedGetResourceMetric

    AbleToScale True means the HPA found the Deployment and could scale it if it wanted. ScalingActive False with FailedGetResourceMetric means it cannot decide whether to.

    This pair is worth memorising, because it rules out the two things people check first: the target reference is fine, and RBAC is fine.

    bash Example session
    kubectl --context cka1001 -n guestbook describe hpa guestbook-ui | sed -n '/Conditions/,/Events/p' | head -8Conditions:  Type           Status  Reason                   Message  ----           ------  ------                   -------  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container guestbook-ui of Pod guestbook-ui-5d6468fd55-cj47cEvents:

    Expected resultScalingActive False with FailedGetResourceMetric.

    Success conditionYou can distinguish "cannot find the target" from "cannot compute a metric".

  3. The Deployment has no requests

    One command ends the investigation:

    {}

    The container declares no resources at all. And a CPU-utilisation HPA computes usage as a percentage of requests. With no requests there is no denominator, so there is no percentage - not zero, undefined.

    This is also why the fix is never "install metrics-server". metrics-server is working; it is reporting usage correctly. The missing piece is in the Pod spec.

    bash Example session
    kubectl --context cka1001 -n guestbook get deploy guestbook-ui -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'{}

    Expected result{} - an empty resources block.

    Success conditionYou can point at the empty field rather than guessing.

  4. The same HPA, on a Deployment that declares requests

    A Deployment with requests.cpu: 50m, autoscaled the same way, reads:

    cpu: 2%/60%

    A real number, and the conditions flip: ScalingActive True / ValidMetricFound, ScalingLimited False / DesiredWithinRange. The HPA is now making an actual decision each interval and concluding, correctly, that one replica is enough.

    2% of 50m is about 1 millicore - the container is idle. Also worth noting: because the percentage is of *requests*, a Pod with a very small request can show hundreds of percent while using almost nothing in absolute terms. The request is a promise you made, not a measurement.

    bash Example session
    kubectl --context cka1001 apply -f - <<'YAML'apiVersion: apps/v1kind: Deploymentmetadata:  name: cpu-demo  namespace: hpa-demospec:  replicas: 1  selector:    matchLabels: { app: cpu-demo }  template:kubectl --context cka1001 -n hpa-demo rollout status deploy/cpu-demo --timeout=120sWaiting for deployment "cpu-demo" rollout to finish: 0 of 1 updated replicas are available...deployment "cpu-demo" successfully rolled outkubectl --context cka1001 -n hpa-demo autoscale deploy cpu-demo --cpu=60% --min=1 --max=4horizontalpodautoscaler.autoscaling/cpu-demo autoscaledkubectl --context cka1001 -n hpa-demo get hpa cpu-demoNAME       REFERENCE             TARGETS       MINPODS   MAXPODS   REPLICAS   AGEcpu-demo   Deployment/cpu-demo   cpu: 2%/60%   1         4         1          64skubectl --context cka1001 -n hpa-demo describe hpa cpu-demo | sed -n '/Conditions/,/Events/p' | head -7Conditions:  Type            Status  Reason              Message  ----            ------  ------              -------  AbleToScale     True    ReadyForNewScale    recommended size matches current size  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable rangeEvents:

    Expected resultA real utilisation percentage, and three healthy conditions.

    Success conditionTARGETS shows a number.

Troubleshooting

Official sources