About the Latency Spike Playbook
Latency spikes are harder than outages: everything is up, dashboards are mostly green, and yet p99 tripled. The cause is usually one slow dependency — a database missing an index, a saturated connection pool, a cache that started missing, GC pauses — amplified through every service that waits on it.
This playbook structures the hunt: confirm the spike is real and locate it (p50 vs p95 vs p99 tells you whether it is systemic or tail-only), then walk the dependency chain — database, cache, network, downstream APIs — with the specific signals that implicate each.
Frequently asked questions
What does it mean when p99 spikes but p50 is flat?
A tail problem: most requests are fine, a subset is very slow. Typical causes are lock contention, GC pauses, connection pool exhaustion (most requests get a pooled connection instantly; the unlucky ones wait), a cold cache shard, or one slow replica behind a load balancer. Flat p50 rules out systemic slowdown — hunt for the minority path.
How do I find which dependency is causing the latency?
Distributed traces answer this directly — sort spans by self-time on slow traces. Without tracing: compare each dependency's own latency metrics against the spike window (DB query time, cache hit ratio, downstream API p99). The dependency whose latency moved first, before the others, is usually the origin; everything downstream of it moved as a consequence.
Why did latency spike with no deploy and no traffic increase?
The usual suspects: a data-shape change (a query plan flipped after autovacuum/statistics, a hot key appeared in cache), an expiring cache or DNS TTL, a noisy neighbor (CPU steal on shared nodes), certificate or connection churn after a dependency restart, or background work (backups, batch jobs, log rotation) colliding with peak traffic. Check the dependency's change history, not just your own.
Need this managed for you, not just automated?
We're also a hands-on DevOps consultancy — Kubernetes, CI/CD, and cloud infrastructure.