Prometheus Targets and Control Plane Scraping

Observability / Guide 38 of 46 / Intermediate

• Kubernetes1.36.4
• Chartkube-prometheus-stack 88.5.3
• Prometheus Operatorv0.93.1
• TimeAbout 16 min
• Reviewed22 August 2026

Written against the versions above. Metric names are stable across these versions; the numbers in the output are from this cluster and yours will differ.

Before you start

• The PromQL guide, which is where the four failing jobs were found.
• SSH to a control plane node, for the one command that proves the cause.

1. 1

   The scrape really is failing

   Before explaining it away, confirm it. up{job="kube-proxy"} returns a sample with value 0 - Prometheus tried, and the scrape did not succeed. This is not a missing metric or a mislabelled target.

   Also worth noticing: the Service the chart expects for kube-proxy does not exist under that name in this release. Chasing that is a dead end, and it is the kind of detour that costs an hour. The metric is the evidence; the Service name is not.

   $kubectl --context cka1001 -n monitoring exec promq -- curl -sG http://kps-kube-prometheus-stack-prometheus:9090/api/v1/query --data-urlencode 'query=up{job="kube-proxy"}'{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","endpoint":"http-metrics","instance":"192.168.0.178:10249","job":"kube-proxy","namespace":"kube-system","pod":"kube-proxy-wpjgh","service":"kps-kube-prometheus-stack-kube-proxy"},"value":[1787345658.752,"0"]},{"metric":{"__name__":"up","endpoint":"http-metrics","instance":"192.168.0.177:10249","job":"kube-proxy","namespace":"kube-system","pod":"kube-proxy-tmb4l","service":"kps-kube-prometheus-stack-kube-proxy"},"value":[1787345658.752,"0"]},{"metric":{"__name__":"up","endpoint":"http-metrics","instance":"192.168.0.175:10249","job":"kube-proxy","namespace":"kube-system","pod":"kube-proxy-mzbdk","service":"kps-kube-prometheus-stack-kube-proxy"},"value":[1787345658.752,"0"]},{"metric":{"__name__":"up","endpoint":"http-metrics","instance":"192.168.0.176:10249","job":"kube-proxy","namespace":"kube-system","pod":"kube-proxy-bxs9g","service":"kps-kube-prometheus-stack-kube-proxy"},"value":[1787345658.752,"0"]}]}}$kubectl --context cka1001 -n monitoring get svc kps-kube-prometheus-stack-kube-proxy -o custom-columns=NAME:.metadata.name,PORTS:.spec.ports[*].portError from server (NotFound): services "kps-kube-prometheus-stack-kube-proxy" not found[exit 1]

   Expected resultA 0 for the kube-proxy target, and a NotFound for the guessed Service name.

   Success conditionYou have confirmed the scrape fails rather than assuming it.

2. 2

   Ask the node, not the cluster

   This is the whole guide in one command. ss -ltn on the control plane shows what is listening and on which address:

   127.0.0.1:2381    etcd metrics
   127.0.0.1:10249   kube-proxy metrics
   127.0.0.1:10259   kube-scheduler

   127.0.0.1, not 0.0.0.0. Those ports are reachable only from processes on that host's loopback interface. Prometheus runs in a Pod with its own network namespace, so its localhost is not the node's localhost. The connection is refused before it reaches anything.

   This is a deliberate kubeadm default, not a misconfiguration. Control plane metrics endpoints are unauthenticated, so binding them to loopback keeps them off the network until an operator decides otherwise.

   $sudo -n ss -ltn | grep -E '127.0.0.1:(10249|10257|10259|2381)'LISTEN 0      4096       127.0.0.1:2381       0.0.0.0:*LISTEN 0      4096       127.0.0.1:10249      0.0.0.0:*LISTEN 0      4096       127.0.0.1:10259      0.0.0.0:*LISTEN 0      4096       127.0.0.1:10257      0.0.0.0:*

   Expected resultThree or four listeners, all bound to 127.0.0.1.

   Success conditionYou can see the loopback address in the output rather than being told about it.

3. 3

   Where the setting lives

   For kube-proxy the value is in a ConfigMap, not a file on disk - which is why looking for /var/lib/kube-proxy/config.conf on the host finds nothing on a kubeadm cluster. The field is metricsBindAddress, and here it is empty:

   metricsBindAddress: ""

   Empty means default, and the default is 127.0.0.1:10249. To expose it you set it to 0.0.0.0:10249 and restart the kube-proxy Pods. The scheduler, controller manager and etcd have equivalent flags in their static Pod manifests under /etc/kubernetes/manifests/.

   Whether you should is a real decision, not a formality. Changing it puts unauthenticated metrics on the node's network. On a lab, fine. On anything shared, bind it and then restrict it - a NetworkPolicy, or a firewall rule that only admits the Prometheus Pods.

   $kubectl --context cka1001 -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep metricsBindAddressmetricsBindAddress: ""

   Expected resultAn empty metricsBindAddress, which is what the loopback binding comes from.

   Success conditionYou can point at the exact setting rather than describing the symptom.

4. 4

   What to do about the alerts

   The chart ships alerting rules that fire on these targets, so left alone this becomes four permanently red alerts - and a permanently red alert trains people to ignore the dashboard. Three defensible options:

   Expose the endpoints and keep the alerts. Correct if you want the metrics, and you accept the exposure and restrict it.

   Disable the scrape jobs you are not going to fix: --set kubeProxy.enabled=false --set kubeScheduler.enabled=false and so on. Honest, and the dashboard stays green because there is nothing pretending to be broken.

   Leave it and document it. The worst option, but common - and if you pick it, write down why, because the next person will spend an afternoon on it.

   Confirm the rules exist before deciding: 35 PrometheusRule objects arrive with the chart, and several are about exactly these targets.

   $kubectl --context cka1001 -n monitoring get prometheusrules --no-headers | head -8kps-kube-prometheus-stack-alertmanager.rules                      3m27skps-kube-prometheus-stack-config-reloaders                        3m27skps-kube-prometheus-stack-etcd                                    3m27skps-kube-prometheus-stack-general.rules                           3m27skps-kube-prometheus-stack-k8s.rules.container-cpu-usage-seconds   3m27skps-kube-prometheus-stack-k8s.rules.container-memory-cache        3m27skps-kube-prometheus-stack-k8s.rules.container-memory-rss          3m27skps-kube-prometheus-stack-k8s.rules.container-memory-swap         3m27s$kubectl --context cka1001 -n monitoring get prometheusrules --no-headers | wc -l35

   Expected resultA list of rule groups and a total. The etcd and kube-proxy groups are the ones that will fire.

   Success conditionYou know how many rule objects the chart brought and where the noise comes from.

Troubleshooting

• You set metricsBindAddress and the target is still down.

  Why: kube-proxy read the ConfigMap at startup. Editing it does not restart anything.

  Fix:kubectl -n kube-system rollout restart ds/kube-proxy, then re-check up{job="kube-proxy"} after a scrape interval.

• etcd stays down after exposing its metrics port.

  Why: etcd metrics are served over TLS with client certificates. Reaching the port is only half of it.

  Fix:The chart expects a Secret with the etcd client cert and key, referenced by kubeEtcd.serviceMonitor.caFile and friends. Without it the scrape fails the handshake, not the connection.

• ss is not installed on the node.

  Why: A minimal image without iproute2.

  Fix:sudo netstat -ltnp if it exists, or read it from inside the container namespace: sudo crictl inspect <id> | grep -i port. Installing iproute2 on a node to debug it is also perfectly reasonable.

Official sources

• kube-proxy configuration reference
• Control plane metrics
• kubeadm static Pod manifests