Generate Incident Response Runbooks

Incident Runbook Templates provides adaptable procedures for detecting, triaging, mitigating, resolving, and communicating operational incidents. It includes separate procedures for a service outage and a PostgreSQL database incident. The overall structure includes impact assessment, alerts, initial investigation, mitigation, root-cause work, resolution, verification, rollback, communication, and escalation.

The service example uses a payment-processing scenario to connect symptoms with investigation paths: complete failure, high latency, partial errors, and traffic surges. It includes deployment history, pod logs, database connection metrics, dependency status, feature flags, scaling, and rate-limit examples. Its names, thresholds, financial amounts, impact figures, and resolution messages are illustrative rather than observations about a real incident.

Use these templates to create a service-specific response guide, document recovery procedures, establish escalation paths, or onboard on-call engineers. They can also organize an active response when the actual service context and approved authority are already established.

Do not run the sample mutations against an unspecified environment. First bind the service, namespace, cluster context, database, owners, evidence sources, response authority, and rollback revision. Read-only triage precedes changes; mitigation requires an approved procedure for the actual service. The referenced smoke-test and database-rollback scripts are not bundled, and their names do not prove that they exist in a target project.

Inputs include the service topology, alert sources, dashboards, severity definitions, recovery options, contact roles, and validated operational commands. Outputs are a scoped runbook, an escalation matrix, verification and rollback steps, and communication templates. The severity table distinguishes complete outage or data loss, major degradation, minor impact, and minimal impact; teams should bind response targets to their own operating agreement.

The database template covers connection-pool exhaustion, replication lag, locks, and disk pressure. It supplies this read-only lock inspection query:

SELECT * FROM pg_locks WHERE NOT granted;

Connection inspection is separated from termination. The source prohibits bulk termination and requires an exact database, process identifier, owner, and approved recovery step. It also warns that VACUUM FULL requires an exclusive lock and extra space, so it is not a generic disk-emergency remedy. Escalation to the database owner remains part of the procedure.

Communication templates cover the first notification, progress updates, and resolution. Fill them with observed impact, actions, and validated recovery evidence rather than copying sample percentages or claiming an unverified resolution. The guide calls for stakeholder updates, early escalation, and postmortems.

Examples use Kubernetes, PostgreSQL, Prometheus, Grafana, Sentry, Slack, PagerDuty, and an external payment dependency. Their endpoints and scripts are placeholders to replace with verified service resources. Review runbooks after incidents, exercise them through game days, document assumptions, link dashboards, and check that every recovery step worked.

These templates support on-call engineers and service owners building response procedures that remain readable under pressure and connect operational actions to verification and communication.