Prometheus Alerting Rules That Don't Wake You Up at 3am
Alert fatigue is real. Here are the rules we use in production: symptom-based alerts, multi-window burn rates, and how to tune thresholds without guessing.
The Alert Fatigue Problem
The on-call rotation at a company I worked with received 200+ alerts per day. After two weeks, the team had developed a "Pavlovian ignore" response — they acknowledged alerts without reading them. A real outage went unnoticed for 47 minutes because it looked like the usual noise.
The problem wasn't their infrastructure. It was their alerting philosophy.
Principle 1: Alert on Symptoms, Not Causes
Bad alert: CPU > 80% for 5 minutes Good alert: Error rate > 1% for 5 minutes
CPU at 80% might be fine (batch job running), might be terrible (service about to fall over), or might be expected (post-deploy warmup). It requires human interpretation.
Error rate > 1% means users are experiencing errors right now. That always requires action.
The distinction: cause-based alerts require engineers to interpret, symptom-based alerts tell engineers what to fix.
``yaml
# Bad — cause-based
expr: cpu_usage_percent > 80
for: 5m
# Good — symptom-based
- alert: HighErrorRate
Principle 2: Use Multi-Window Burn Rates
Single-window alerts are noisy. A 1-minute spike triggers an alert, wakes someone up, and resolves before they've opened their laptop.
Multi-window burn rates solve this. From the Google SRE Book:
yaml
# Alert when error budget is burning too fast
# Short window catches fast-burn incidents
# Long window catches slow-burn degradation
- alert: ErrorBudgetBurnRate
expr: |
(
rate(http_requests_total{status=~"5.."}[1h])
/ rate(http_requests_total[1h])
) > 14.4 * 0.001 # 14.4x burn rate = exhaust monthly budget in 2 days
and
(
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m])
) > 14.4 * 0.001
for: 2m
labels:
severity: critical
annotations:
summary: "High error budget burn rate"
`The
condition requires both the 1h trend AND the 5m trend to exceed the threshold. A brief spike passes the 5m check but fails the 1h check — no alert. A sustained degradation passes both — alert fires.Principle 3: Use
for Duration WiselyThe
for clause waits N minutes before firing. This absorbs transient spikes.`yaml
# No 'for' — fires on the first evaluation
- alert: DatabaseDown
expr: up{job="postgresql"} == 0
# Good: we want to know immediately if DB is unreachable# 5m 'for' — waits for sustained condition
- alert: HighLatency
expr: histogram_quantile(0.99, rate(http_duration_bucket[5m])) > 2
for: 5m
# Good: a 2-second P99 for 5+ minutes is a real problem
`Rules:
- Availability alerts (service down, database unreachable):
or for: 1m
Latency/error rate alerts: for: 5m to absorb spikes
Capacity alerts (disk filling up): for: 30m or longer
Principle 4: Tier Your Alerts
Not everything needs to wake someone up at 3am.
`yaml
labels:
severity: critical # Page immediately, any time
severity: warning # Slack notification, business hours response
severity: info # Dashboard annotation, no human action needed
`yaml
# Critical: user impact, needs immediate action
- alert: ServiceDown
expr: up == 0
labels:
severity: critical# Warning: trending toward problem, fix during business hours
- alert: DiskFillingUp
expr: disk_free_percent < 20
for: 30m
labels:
severity: warning# Info: notable event, no action needed
- alert: DeploymentCompleted
expr: kube_deployment_status_replicas_updated == kube_deployment_spec_replicas
labels:
severity: info
`Only
goes to PagerDuty. warning goes to Slack. info goes to a low-noise channel or nowhere.Principle 5: Include Runbooks in Annotations
Every critical alert should link to a runbook. The engineer at 3am should not have to think — they should follow steps.
`yaml
- alert: ConnectionPoolExhausted
expr: hikaricp_connections_pending > 0
for: 2m
labels:
severity: critical
annotations:
summary: "HikariCP connection pool exhausted on {{ $labels.instance }}"
description: |
{{ $value }} requests waiting for a DB connection.
Active: {{ query "hikaricp_connections_active" | first | value }}
Max: {{ query "hikaricp_connections_max" | first | value }}
runbook_url: "https://wiki.company.com/runbooks/connection-pool-exhausted"
`The annotation includes current values from Prometheus so the engineer immediately sees the scope of the problem without needing to open Grafana.
Tuning Thresholds Without Guessing
The biggest source of noisy alerts: arbitrary thresholds (
, latency > 1s).Better approach: use historical data to set thresholds.
`promql
# What is P99 latency during normal operations?
histogram_quantile(0.99, rate(http_duration_bucket[30d]))# What is the 95th percentile of CPU usage over 30 days?
quantile_over_time(0.95, cpu_usage_percent[30d])
`Set your alert threshold at 2-3× the normal P99. If normal P99 latency is 200ms, alert at 600ms. This way you're alerted when something genuinely abnormal happens, not when there's normal variation.
The Alert Audit Process
Every quarter, review your alerts:
Which alerts fired most? High-frequency + low-action-rate = noise, tune or delete
Which alerts were acknowledged without action? Same conclusion
Were there incidents NOT caught by alerts? Add coverage sql
-- In your alertmanager database or Grafana
SELECT alert_name, COUNT(*) as fires, AVG(duration_minutes) as avg_duration
FROM alert_history
WHERE resolved_without_action = true
GROUP BY alert_name
ORDER BY fires DESC;
``Aim for: every alert that fires requires human action 90%+ of the time. Below 70% = too noisy, needs tuning.