CertGrid CertGrid
Hands-on Lab·Kubernetes and Cloud Native Associate

Installing Prometheus on Kubernetes

Install kube-prometheus-stack with Helm and watch all four node-exporters go into CrashLoopBackOff. The cause is one line in the logs and one line of ss output: the DaemonSet uses hostNetwork, and the host already had something on that port. Both fixes are shown, with the output of each.

Observability Guide 36 of 46 Beginner

Written against the versions above. Chart and image versions move quickly. The behaviour in this guide - hostNetwork sharing the node's port space - is a property of the DaemonSet, not of a release.

kube-prometheus-stack installs across all four nodes: one Prometheus, one Grafana, one Alertmanager, and a node-exporter on every node.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. Nothing is watching yet

    A fresh cluster collects almost nothing. metrics-server may give you kubectl top, but that is a live snapshot with no history: you cannot ask it what happened twenty minutes ago, and it stores no series you can query.

    Prometheus is the other half. It scrapes HTTP endpoints on a schedule, keeps the samples as time series, and lets you ask questions across them. Start by confirming there is nothing in the namespace you are about to fill.

    bash Example session 
    kubectl --context cka1001 get pods -n monitoringNo resources found in monitoring namespace.kubectl --context cka1001 top nodesNAME             CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)cka1001          130m         6%       1725Mi          52%cka1001-node01   51m          2%       990Mi           30%cka1001-node02   58m          2%       1125Mi          34%cka1001-node03   52m          2%       807Mi           24%

    Expected resultAn empty namespace, and four nodes with memory to spare. Note the memory percentages - you will want to compare them afterwards.

    Success conditionNo resources found in monitoring namespace.

  2. Add the chart

    kube-prometheus-stack is the community chart that installs Prometheus, Alertmanager, Grafana, kube-state-metrics and node-exporter together, plus the Prometheus Operator that manages them through custom resources.

    It is worth knowing why that bundle exists. Prometheus on its own scrapes targets you configure by hand. The Operator lets you declare targets as ServiceMonitor objects instead, so adding a scrape target becomes a normal Kubernetes apply rather than a config-file edit and reload.

    bash Example session 
    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts"prometheus-community" already exists with the same configuration, skippinghelm repo update prometheus-communityHang tight while we grab the latest from your chart repositories......Successfully got an update from the "prometheus-community" chart repositoryUpdate Complete. ⎈Happy Helming!⎈helm search repo prometheus-community/kube-prometheus-stack --versions | head -3NAME                                      	CHART VERSION	APP VERSION	DESCRIPTIONprometheus-community/kube-prometheus-stack	88.5.3       	v0.93.1    	kube-prometheus-stack collects Kubernetes manif...prometheus-community/kube-prometheus-stack	88.5.2       	v0.93.1    	kube-prometheus-stack collects Kubernetes manif...level=WARN msg="repo is corrupt or missing" repo=traefik error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\traefik-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=grafana error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\grafana-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=rancher-latest error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\rancher-latest-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=argo error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\argo-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=portainer error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\portainer-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=aqua error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\aqua-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=cilium error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\cilium-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=kiali error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\kiali-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=istio error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\istio-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=metallb error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\metallb-index.yaml: The system cannot find the file specified."level=WARN msg="repo is corrupt or missing" repo=jetstack error="open C:\\Users\\Sunil\\AppData\\Local\\Temp\\helm\\repository\\jetstack-index.yaml: The system cannot find the file specified."Error: unable to write table output: write /dev/stdout: The pipe has been ended.

    Expected resultThe chart version and the app version it ships. Pin these in anything real - the chart moves faster than Kubernetes does.

    Success conditionhelm search repo lists a kube-prometheus-stack version.

  3. Install it

    Two values are set deliberately. retention=6h keeps the disk small for a lab; the chart's default is ten days. The memory request keeps Prometheus schedulable on a 4 GB node - leave the default and it may sit Pending on a small cluster, which is a scheduling problem wearing an observability costume.

    bash Example session 
    kubectl --context cka1001 create namespace monitoringnamespace/monitoring createdhelm --kube-context cka1001 install kps prometheus-community/kube-prometheus-stack --namespace monitoring --set grafana.enabled=true --set prometheus.prometheusSpec.retention=6h --set prometheus.prometheusSpec.resources.requests.memory=400MiNAME: kpsLAST DEPLOYED: Sat Aug 22 02:18:37 2026NAMESPACE: monitoringSTATUS: deployedREVISION: 1DESCRIPTION: Install completeTEST SUITE: NoneNOTES:kube-prometheus-stack has been installed. Check its status by running:  kubectl --namespace monitoring get pods -l "release=kps" Get Grafana 'admin' user password by running:   kubectl --namespace monitoring get secrets kps-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo Access Grafana local instance:   export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kps" -oname)  kubectl --namespace monitoring port-forward $POD_NAME 3000 Get your grafana admin user password by running:   kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo  Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

    Expected resultHelm reports the release as deployed. That means the objects were accepted, not that the Pods are healthy - the next step is where that shows.

    Success conditionSTATUS: deployed.

  4. What it created, and what broke

    Read this output carefully rather than waiting for it to settle. The operator, kube-state-metrics and Grafana come up. All four node-exporters are in CrashLoopBackOff, and they got there within forty seconds, which rules out anything slow or resource-related.

    Four out of four is the useful detail. A crash on one node suggests something about that node; a crash on every node suggests something about the workload, or something true of all four hosts.

    bash Example session 
    kubectl --context cka1001 -n monitoring get podsNAME                                                    READY   STATUS              RESTARTS      AGEalertmanager-kps-kube-prometheus-stack-alertmanager-0   0/2     PodInitializing     0             20skps-grafana-8497974b5-rq8dc                             0/3     ContainerCreating   0             40skps-kube-prometheus-stack-operator-7b847d8968-hnsj2     1/1     Running             0             40skps-kube-state-metrics-659896646d-bqwdv                 1/1     Running             0             40skps-prometheus-node-exporter-22sl7                      0/1     CrashLoopBackOff    2 (17s ago)   40skps-prometheus-node-exporter-78hpk                      0/1     CrashLoopBackOff    2 (12s ago)   40skps-prometheus-node-exporter-nxhj5                      0/1     CrashLoopBackOff    2 (13s ago)   40skps-prometheus-node-exporter-pvwsz                      0/1     CrashLoopBackOff    2 (17s ago)   40sprometheus-kps-kube-prometheus-stack-prometheus-0       0/2     Init:0/1            0             20skubectl --context cka1001 -n monitoring get svcNAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGEalertmanager-operated                    ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   23skps-grafana                              ClusterIP   10.111.173.89    <none>        80/TCP                       43skps-kube-prometheus-stack-alertmanager   ClusterIP   10.104.15.195    <none>        9093/TCP,8080/TCP            43skps-kube-prometheus-stack-operator       ClusterIP   10.102.245.192   <none>        443/TCP                      43skps-kube-prometheus-stack-prometheus     ClusterIP   10.111.140.94    <none>        9090/TCP,8080/TCP            43skps-kube-state-metrics                   ClusterIP   10.106.192.114   <none>        8080/TCP                     43skps-prometheus-node-exporter             ClusterIP   10.101.129.169   <none>        9100/TCP                     43sprometheus-operated                      ClusterIP   None             <none>        9090/TCP                     23skubectl --context cka1001 get crd -o name | grep monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/prometheusagents.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.comcustomresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com

    Expected resultA DaemonSet Pod per node, all failing, plus the CRDs the Operator installs - prometheuses, servicemonitors, prometheusrules and the rest.

    Success conditionYou can see CrashLoopBackOff on every node-exporter Pod.

  5. The logs say it in one line

    Skip the collector list - node-exporter prints every collector it enables at startup, and it is noise here. The last line of each attempt is the whole story:

    listen tcp 0.0.0.0:9100: bind: address already in use

    Not a permission problem, not a missing mount. Something already holds port 9100. And this is a hostNetwork DaemonSet: the Pod does not get its own network namespace, it uses the node's. So "already in use" means already in use *on the host*, by something that has nothing to do with Kubernetes.

    bash Example session 
    kubectl --context cka1001 -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporterNAME                                 READY   STATUS             RESTARTS      AGEkps-prometheus-node-exporter-22sl7   0/1     CrashLoopBackOff   3 (36s ago)   81skps-prometheus-node-exporter-78hpk   0/1     CrashLoopBackOff   3 (30s ago)   81skps-prometheus-node-exporter-nxhj5   0/1     CrashLoopBackOff   3 (29s ago)   81skps-prometheus-node-exporter-pvwsz   0/1     CrashLoopBackOff   3 (36s ago)   81skubectl --context cka1001 -n monitoring logs -l app.kubernetes.io/name=prometheus-node-exporter --tail=6 --prefix=falsetime=2026-08-21T20:49:50.111Z level=INFO source=node_exporter.go:142 msg=unametime=2026-08-21T20:49:50.111Z level=INFO source=node_exporter.go:142 msg=vmstattime=2026-08-21T20:49:50.111Z level=INFO source=node_exporter.go:142 msg=watchdogtime=2026-08-21T20:49:50.111Z level=INFO source=node_exporter.go:142 msg=xfstime=2026-08-21T20:49:50.111Z level=INFO source=node_exporter.go:142 msg=zfstime=2026-08-21T20:49:50.111Z level=ERROR source=node_exporter.go:248 msg="listen tcp 0.0.0.0:9100: bind: address already in use"time=2026-08-21T20:49:51.803Z level=INFO source=node_exporter.go:142 msg=unametime=2026-08-21T20:49:51.803Z level=INFO source=node_exporter.go:142 msg=vmstat

    Expected resultThe same bind: address already in use at the end of each restart attempt.

    Success conditionYou have found the error line rather than guessing from the Pod status.

  6. Ask the host what owns the port

    This is the step that turns a guess into an answer. ss -ltnp lists listening TCP sockets with the process that owns each one - and the answer is a node_exporter that was installed on the host directly, as a systemd unit, long before this chart existed.

    Two exporters, one port, and the Pod loses because the host process got there first. Nothing about the cluster is broken.

    bash Example session 
    sudo -n ss -ltnp | grep -w 9100LISTEN 0      4096               *:9100             *:*    users:(("node_exporter",pid=847,fd=3))systemctl is-active node_exporter prometheus-node-exporter 2>&1 | head -3activeinactivesudo -n systemctl show -p Description,FragmentPath node_exporter | head -3Description=Node ExporterFragmentPath=/etc/systemd/system/node_exporter.service

    Expected resultA pid holding *:9100, and a systemd unit that explains where it came from.

    Success conditionYou can name the process on 9100 and the unit file behind it.

  7. Move the chart, not the host

    There are two honest fixes and the choice matters.

    Stop the host exporter if it is redundant - two processes exporting the same metrics is waste, and the DaemonSet is the one Kubernetes can manage, upgrade and reschedule.

    Move the chart's exporter to another port if the host one is someone else's: monitored by a system that predates the cluster, or owned by a team who did not agree to this install. That is the assumption here, so 9101 it is.

    What you must not do is delete the DaemonSet and call the stack installed. Without node-exporter you have no CPU, memory, disk or network metrics for any node - which is most of what people install Prometheus for.

    bash Example session 
    helm --kube-context cka1001 upgrade kps prometheus-community/kube-prometheus-stack --namespace monitoring --reuse-values --set prometheus-node-exporter.service.port=9101 --set prometheus-node-exporter.service.targetPort=9101Release "kps" has been upgraded. Happy Helming!NAME: kpsLAST DEPLOYED: Sat Aug 22 02:21:02 2026NAMESPACE: monitoringSTATUS: deployedREVISION: 2DESCRIPTION: Upgrade completeTEST SUITE: NoneNOTES:kube-prometheus-stack has been installed. Check its status by running:  kubectl --namespace monitoring get pods -l "release=kps" Get Grafana 'admin' user password by running:   kubectl --namespace monitoring get secrets kps-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo Access Grafana local instance:   export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kps" -oname)  kubectl --namespace monitoring port-forward $POD_NAME 3000 Get your grafana admin user password by running:   kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo  Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.kubectl --context cka1001 -n monitoring rollout status ds/kps-prometheus-node-exporter --timeout=120sdaemon set "kps-prometheus-node-exporter" successfully rolled outkubectl --context cka1001 -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporterNAME                                 READY   STATUS    RESTARTS   AGEkps-prometheus-node-exporter-66k7t   1/1     Running   0          11skps-prometheus-node-exporter-hhg9v   1/1     Running   0          11skps-prometheus-node-exporter-hvgzr   1/1     Running   0          11skps-prometheus-node-exporter-wmq2w   1/1     Running   0          11ssudo -n ss -ltnp | grep -E ':(9100|9101)'LISTEN 0      4096               *:9101             *:*    users:(("node_exporter",pid=137898,fd=4))LISTEN 0      4096               *:9100             *:*    users:(("node_exporter",pid=847,fd=3))

    Expected resultFour Pods Running, and ss showing both exporters: the host one still on 9100, the chart one now on 9101.

    Success condition4/4 node-exporter Pods Running, and two distinct pids on two ports.

  8. Or remove the host exporter, if it is yours to remove

    The previous step took the cautious option. Here is the other one, with the same rigour, because on this cluster the host exporter really was redundant - a hand-installed systemd unit from September, duplicating exactly what the DaemonSet collects.

    systemctl disable --now stops it and removes the boot symlink. It does not delete the binary or the unit file, so systemctl enable --now node_exporter puts it back - which is the right level of destructiveness for something you did not install yourself.

    With 9100 free, the chart can have its default port back. All four Pods roll out, and ss now shows the container holding 9100 instead of the host process.

    Then the check that matters: count(up{job="node-exporter"} == 1) returns 4. All four nodes are being scraped. And count(up == 0) is still 7 - unchanged, because those are the control-plane targets bound to localhost, which have nothing to do with this.

    Which option to pick is a judgement about ownership, not about correctness. Two exporters on two ports works and wastes a little memory. One exporter on the standard port is cleaner and matches every dashboard's assumptions. What decides it is whether something you do not control was already scraping 9100 - and if you cannot answer that, take the cautious option first.

    bash Example session 
    sudo -n systemctl disable --now node_exporter && echo disabled on $(hostname)disabled on cka1001Removed '/etc/systemd/system/multi-user.target.wants/node_exporter.service'.sudo -n ss -ltnp | grep -w 9100 || echo "nothing on 9100 now"nothing on 9100 nowhelm --kube-context cka1001 upgrade kps prometheus-community/kube-prometheus-stack --namespace monitoring --reuse-values --set prometheus-node-exporter.service.port=9100 --set prometheus-node-exporter.service.targetPort=9100Release "kps" has been upgraded. Happy Helming!NAME: kpsLAST DEPLOYED: Sat Aug 22 10:17:58 2026NAMESPACE: monitoringSTATUS: deployedREVISION: 3kubectl --context cka1001 -n monitoring rollout status ds/kps-prometheus-node-exporter --timeout=180sdaemon set "kps-prometheus-node-exporter" successfully rolled outkubectl --context cka1001 -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporterNAME                                 READY   STATUS    RESTARTS   AGEkps-prometheus-node-exporter-k2xvn   1/1     Running   0          12skps-prometheus-node-exporter-llzk8   1/1     Running   0          10skps-prometheus-node-exporter-qr2dh   1/1     Running   0          11skps-prometheus-node-exporter-qwf76   1/1     Running   0          9skubectl --context cka1001 -n monitoring exec promq -- curl -sG http://kps-kube-prometheus-stack-prometheus:9090/api/v1/query --data-urlencode 'query=count(up{job="node-exporter"} == 1)'{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1787374165.866,"4"]}]}}

    Expected resultThe unit disabled, 9100 free, four Pods rolled out on the default port, and four node-exporter targets up.

    Success conditioncount(up{job="node-exporter"} == 1) returns the number of your nodes.

Troubleshooting

Official sources

← All KCNA guides