Playbook

Incident Triage Playbook

Classify severity quickly and run the first checks with confidence.

Sev 1: full outage, data loss, safety/security breach, or SLO breach in progress. Ack within 5 min.

Sev 2: major feature down, significant latency or error rate. Ack within 15 min.

Sev 3: partial degradation, limited user impact. Ack within 1 hr.

Sev 4: minor issue or cosmetic impact. Handle next business day.

Sev 5: no user impact, informational alert only. Handle async.

Confirm scope: regions, tenants, endpoints, and user tiers.

Establish a timeline: first alert timestamp, first user report, last known good state.

Check recent deploys, config changes, and feature flags.

Verify dependencies: DB, cache, queues, and third-party APIs.

Check SLO error budget burn rate, not just raw error rate.

Mitigate first: rollback, scale, or disable risky features.

Do not start root cause analysis until the incident is stabilized.

Set a 30-minute decision checkpoint — if not stabilized, escalate severity.

Communicate: update status page and incident channel.

Assign roles: incident commander, comms, and scribe.

Schedule a blameless postmortem within 48–72 hours.

Capture full timeline, contributing factors, and action items with owners and due dates.

Update runbooks, dashboards, and alert thresholds based on gaps found.

Recent cluster events

kubectl get events --sort-by='.lastTimestamp' -A | tail -20

Pod restart counts across namespaces

kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'

Current 5xx error rate (Prometheus)

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Need this managed for you, not just automated?

We're also a hands-on DevOps consultancy — Kubernetes, CI/CD, and cloud infrastructure.