SLOs, SLIs and Error Budgets

Observability / Guide 41 of 46 / Intermediate

• Kubernetes1.36.4
• Cluster4 nodes
• Runtimecontainerd 2.2.6
• CNICalico v3.32.1
• TimeAbout 19 min

Recording rules and the /api/v1/query endpoint are stable Prometheus. The PrometheusRule CRD comes from the Prometheus Operator, which runs in this cluster's monitoring namespace.

• Prometheus3.14.0
• kube-prometheus-stackinstalled in the monitoring namespace

This guide includes

Use this when an SLO has to become something the system can answer rather than a number in a document. This matters because an availability target is only useful once it is a named query with an error budget you can watch shrink.

• choosing an SLI as an explicit ratio, rather than assuming availability is measured for you
• giving that expression a name with a recording rule, so it is not a line in someone's browser history
• computing the error budget as the distance between the SLI and a 99.5% target
• checking for the third pillar and finding no OpenTelemetry or Jaeger CRDs installed

Before you start

• The Prometheus install guide, since every query here goes to that server.
• The PromQL guide - rate() and label matchers are assumed.

1. 1

   An SLI is a ratio you have to choose

2. 2

   A recording rule gives it a name

3. 3

   The error budget is the distance to the target

4. 4

   The third pillar you do not have

Official sources

• Recording rules
• Metric and label naming
• Alerting on SLOs
• OpenTelemetry concepts