Playbook

Error Rate Playbook

Track top error sources and quickly decide on rollback or mitigation.

Rank endpoints by error rate and request volume.

Check if a single route or method dominates the spike.

Compare error types (4xx vs 5xx) to narrow root cause.

Review the last 24h deploys, feature flags, and config changes.

Look for schema migrations or dependency upgrades.

Check rollout status and region-by-region impact.

Disable recent flags or roll back to last stable build.

Validate error rate recovery within 5-10 minutes.

Document the change window and mitigation outcome.

Top endpoints by 5xx

# Prometheus example
topk(5, sum(rate(http_requests_total{status=~"5.."}[5m])) by (route))

Top error codes in logs

# Example with jq
cat app.log | jq -r '.status' | sort | uniq -c | sort -nr | head

Check recent deploys

kubectl rollout history deployment/api