Hands-on Lab·Kubernetes and Cloud Native Associate
Default Alerting Rules and Alert Tuning
A fresh kube-prometheus-stack fires nineteen alerts on a working cluster. One of them is firing on purpose, several are the localhost-binding problem, and the rest are why alert fatigue starts on day one. Then write a rule of your own and watch it load.
Observability Guide 39 of 46 Intermediate
- Kubernetes1.36.4
- Cluster4 nodes
- Runtimecontainerd 2.2.6
- CNICalico v3.32.1
- TimeAbout 19 min
Alert names come from kube-prometheus-stack and change between chart versions. The counts are from this cluster.
- Chartkube-prometheus-stack 88.5.3
- Prometheus Operatorv0.93.1
| Server Name | IP Address | OS | Roles | CPU | RAM | HDD |
|---|---|---|---|---|---|---|
| CKA1001 | 192.168.0.175 | Ubuntu 26.04 LTS | Control Plane Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE01 | 192.168.0.176 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
This guide includes
Use this when a new monitoring install is already firing alerts nobody can act on. This matters because default rules assume a cluster that does not exist yet, and alerts nobody acts on train people to ignore all of them.
- counting what is firing using the
ALERTSseries Prometheus generates for its own rules - seeing the same alerts in Alertmanager, which groups and deduplicates rather than evaluates
- writing a PrometheusRule of your own as a Kubernetes object rather than a config file
- confirming it loaded and is firing by asking the rules API for that group by name
Before you start
- kube-prometheus-stack installed, and the
promqquery Pod from the PromQL guide. - The control-plane targets guide helps - several of these alerts are that problem.
-
Count what is firing
-
Alertmanager has them too
-
A rule of your own
-
It loaded, and it is firing