Generate Incident Response Runbooks

Provides production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication - a fixed nine-section runbook structure (overview/impact, detection/alerts, initial triage, mitigation steps, root cause investigation, resolution procedures, verification/rollback, communication templates, escalation matrix) applied to two worked examples: a service outage runbook and a database incident runbook.

Use this skill when creating incident response procedures, building service-specific runbooks, establishing escalation paths, documenting recovery procedures, responding to active incidents, or onboarding on-call engineers. Do not use it for tasks unrelated to incident runbook templates, or when a different domain or tool is actually needed.

Incident severity levels: SEV1 (complete outage/data loss, 15-minute response), SEV2 (major degradation, 30-minute response), SEV3 (minor impact, 2-hour response), SEV4 (minimal/cosmetic impact, next business day).

The Service Outage Runbook template covers: impact assessment questions (affected customers, traffic percentage, financial implications, blast radius); detection via named alert thresholds and linked dashboards; initial triage in the first 5 minutes using commands like:

kubectl get pods -n payments -l app=payment-service
kubectl rollout history deployment/payment-service -n payments

symptom-to-section classification (all requests failing -> service down, high latency -> database/dependency, partial failures -> code bug, error spike -> traffic surge); mitigation procedures per scenario (rollback via kubectl rollout undo, scaling replicas, killing long-running Postgres queries, enabling a circuit breaker env var, feature-flag disabling, rate limiting, and NetworkPolicy IP blocking); verification via health/error-rate/latency queries and a smoke test script; rollback procedures for deployments, database migrations, and feature flags; an escalation matrix (>15min unresolved SEV1 to engineering manager, suspected data breach to security, >$10k financial impact to finance/legal); and three communication templates (initial notification, status update, resolution notification) with concrete example text.

The Database Incident Runbook template covers a quick-reference command table (check connections, kill query, check replication lag, check locks), connection pool exhaustion queries (pg_stat_activity grouping, idle connection termination), replication lag diagnosis (pg_last_wal_receive_lsn() vs pg_last_wal_replay_lsn() comparison, failover consideration above 60s lag), and disk space handling (df -h, largest-table lookup via pg_total_relation_size, VACUUM FULL).

Best practices: keep runbooks updated after every incident, test them regularly via game days/chaos engineering, always include rollback steps, document the assumptions each step depends on, and link directly to dashboards. Anti-patterns: don't assume responder knowledge ("write for 3 AM brain"), don't skip step verification, don't forget stakeholder communication, don't work alone, and don't skip postmortems.

Built around Kubernetes (kubectl), PostgreSQL (psql/pg_stat_activity), Prometheus queries, Grafana/Sentry dashboards, and Slack/PagerDuty for alerting and communication, with further reading pointers to the Google SRE Book, PagerDuty Incident Response, and Atlassian Incident Management guides.

SRE and on-call engineers building or maintaining incident runbooks for services and databases who want concrete, copy-adaptable command sequences and communication templates rather than starting a runbook from a blank page.