Why P99 Latency is Lying to You (And What to Use Instead)
P99 is the standard. P99 is also misleading. Here's why histograms, P999, and error budgets give a much clearer picture of user experience.
The P99 Problem
Your dashboard shows P99 latency = 250ms. Green. All good.
Meanwhile, 1 in 100 users is waiting 8 seconds for a response. At 1000 RPS, that's 10 users per second having a terrible experience. In a day: 864,000 bad experiences.
P99 told you everything was fine.
Why Percentiles Hide Problems
A percentile is a single number that summarizes a distribution. The problem: performance distributions are not smooth or symmetric.
Consider two services with identical P99 = 250ms:
Service A:
- P50: 50ms, P90: 120ms, P99: 250ms, P99.9: 280ms
- Fast service with a small tail
- P50: 50ms, P90: 120ms, P99: 250ms, P99.9: 12,000ms
- Same P99, but 1 in 1000 requests takes 12 seconds
What to Use Instead
1. P99.9 and P99.99
One extra "9" reveals the outliers:
``yaml
# Prometheus recording rules
expr: histogram_quantile(0.999, rate(http_request_duration_bucket[5m]))
- record: api:request_duration_p9999
If P99 = 250ms but P99.9 = 12s, you have a serious outlier problem.
2. Histograms, Not Summaries
Prometheus offers two metric types for latency: summaries and histograms.
Summary (avoid):
http_request_duration_p99 0.250
`
- Calculated client-side, cannot be aggregated across instances
- P99 of (P99 of instance-1, P99 of instance-2) ≠ P99 of all requests
Histogram (use this):
`
http_request_duration_bucket{le="0.1"} 8472
http_request_duration_bucket{le="0.25"} 9821
http_request_duration_bucket{le="1.0"} 9998
http_request_duration_bucket{le="+Inf"} 10000
`
- Calculated server-side from bucket data
- Fully aggregatable: histogram_quantile() works correctly across instances
3. Error Budgets (SLO-Based)
Instead of: "P99 < 250ms"
Use: "99.9% of requests complete in < 500ms, measured over 28 days"
The 0.1% budget = 40.3 minutes of allowed violations per 28 days.
`yaml
# Alert when burn rate is too high
- alert: ErrorBudgetBurnRate
expr: |
(
rate(http_requests_total{status=~"5.."}[1h]) /
rate(http_requests_total[1h])
) > 0.001 # 0.1% error budget
for: 5m
`The User Experience Test
When reviewing latency metrics, ask: "What does the worst 1 in 1000 users experience?"
If that number is acceptable, you're fine. If it's 10 seconds for a web page load, P99 being green means nothing.
Practical Dashboard Setup
Grafana panels to add:
Latency heatmap (shows full distribution, not just percentiles)
P50 / P99 / P99.9 as separate lines
Error rate (5xx / total)
Apdex score (satisfied + tolerating / total)
``The Apdex score is especially useful for stakeholder communication — it's a single 0-1 score that accounts for both fast and slow responses.
Summary
| Metric | Use Case | Pitfall |
| P99 | General health | Hides outliers |
| P99.9 | SLO definition | Needs high volume to be stable |
| Histogram | Deep analysis | Requires bucket pre-configuration |
| Apdex | Executive reporting | Needs threshold definition |
| Error budget | SLO alerting | Requires 28-day window for stability |