About the Error Rate Playbook
A rising error rate is a symptom with many possible causes, and the fastest way to narrow it is to slice before you dig. Which status codes, which endpoints, which dependency, which version, which zone — one of those dimensions almost always isolates the problem within the first few minutes.
The most useful early split is between 4xx and 5xx. A jump in 5xx means the service is failing; a jump in 4xx usually means something changed upstream — a client deploy, an expired credential, or a new validation rule rejecting traffic that used to pass. They lead in opposite directions and conflating them wastes the time that matters most.
This playbook covers that triage sequence, then the specific patterns worth recognising: errors that track a deployment, errors confined to one instance or zone, errors that follow a dependency's latency, and the retry storms that turn a brief upstream blip into a sustained outage.
Frequently asked questions
Where should I look first when errors spike?
At what changed. The overwhelming majority of error spikes follow a deploy, a configuration change, a feature flag, or a credential rotation. Check the deployment timeline against the error onset before investigating anything else — if they align, roll back and investigate afterwards rather than debugging in production.
How do I tell a client problem from a server problem?
Split by status class first. A 5xx rise means your service is failing and the cause is inside your boundary or in a dependency. A 4xx rise usually means callers changed — a new client version, an expired token, or a validation change rejecting previously accepted input. A 4xx spike from a single caller identifies itself immediately.
What is a retry storm and how do I recognise it?
It is when clients respond to failures by retrying, which increases load on an already struggling service and causes more failures. The signature is that request volume rises as error rate rises, and the service does not recover when the original trigger is removed. The fixes are exponential backoff with jitter, a retry budget capping retries as a fraction of traffic, and circuit breakers.
Should I alert on error rate or error count?
Rate, almost always, because a fixed count is simultaneously too noisy at low traffic and too insensitive at high traffic. The stronger form is a burn-rate alert against an error budget, which fires quickly for a severe spike and slowly for a mild sustained elevation. Alerting on raw counts is a reliable source of pages nobody acts on.
Why do errors appear in only one availability zone?
That pattern points at infrastructure rather than code, since the application version is identical everywhere. Look at a dependency with zonal affinity — a database replica, a cache node, a NAT gateway — or at cross-zone network problems. It can also be a stale endpoint list sending traffic to instances that no longer exist in that zone.
Need this managed for you, not just automated?
We're also a hands-on DevOps consultancy — Kubernetes, CI/CD, and cloud infrastructure.