Playbook

Latency Spike Playbook

Track p95/p99 regression and isolate DB, cache, or network causes quickly.

Confirm p95/p99 impact by endpoint, region, and customer tier.

Compare to baseline and last known good window.

Check error rates, saturation, and queue depths in parallel.

DB: slow queries, locks, connection pool saturation, replication lag.

Cache: hit rate drop, eviction spikes, hot keys, node health.

Network: packet loss, increased RTT, load balancer retries.

Rollback recent deploys or disable new feature flags.

Scale read replicas, cache nodes, or app pods.

Increase timeouts cautiously while fixing root cause.

Track p95/p99 by endpoint

# Prometheus example
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))

Trace slow requests

curl -H "x-request-id: trace-me" -w "time_total=%{time_total}
" -o /dev/null -s https://api.example.com/health

Check network RTT

ping -c 5 api.example.com