Orchestrate Incident Response with SRE Practices

Incident Response Incident Response is a skill for orchestrating multi-agent incident response following modern SRE and Incident Command System (ICS) practices, running specialized agents through five phases so resolution stays fast without sacrificing accuracy, and every incident becomes a blameless learning opportunity.

Phase 1 (detection and triage) classifies severity - P0/SEV-1 for complete outages, security breaches, or data loss requiring an all-hands response, down to P3/SEV-4 for cosmetic issues on a scheduled fix - checks error-budget and SLO status, runs an observability sweep across distributed tracing, metrics, log aggregation, and APM/RUM data, and applies immediate mitigation (traffic throttling, feature-flag kills, circuit breakers, rollback assessment, or scaling). Phase 2 (investigation and root cause) does deep debugging with a Five Whys analysis across stack traces, database locks, network latency, and memory/CPU, a security assessment for DDoS indicators, auth failures, or data exposure via WAF logs and audit trails, and a performance-engineering pass over resource utilization, caching, load balancers, CDN, and autoscaling. Phase 3 (resolution and recovery) implements a minimal viable production fix with a rollback plan and validation criteria, then deploys it via blue-green or canary rollout with staged health checks and configured rollback triggers. Phase 4 (communication and coordination) produces status-page updates, internal engineering detail, an executive summary, and customer-support talking points on a 15-30 minute cadence scaled to severity, plus a customer-impact assessment covering affected segments, failed transactions, SLA violations, and revenue impact. Phase 5 (postmortem and prevention) writes a blameless postmortem (full timeline, systems-focused root cause, what worked, what to improve, owned action items), then enhances monitoring (new alerts, SLI/SLO adjustments, runbook automation, chaos-engineering scenarios) and proposes system hardening (circuit breakers, bulkheads, graceful degradation, capacity planning, dependency reduction).

Use it for an active production incident that needs a structured, multi-phase response with clear roles rather than ad hoc firefighting - it defines an Incident Commander (decision authority), a Technical Lead, a Communications Lead, and subject-matter experts, coordinating through a war room channel, a status page, PagerDuty/Opsgenie alerting, and shared incident documentation. It is not a substitute for having those communication channels and on-call tooling already in place - the workflow assumes they exist and orchestrates through them.

Input is the incident itself (alerts, affected services, initial symptoms). Output is a severity classification, a mitigation and fix, deployment validation, stakeholder communications, and a completed postmortem with tracked action items. Its own success criteria: during the incident, severity classified within 5 minutes, stakeholders updated every 15-30 minutes, and service restored within SLA with no cascading failures; after it, a postmortem within 48 hours, monitoring improvements within a week, and completed runbook updates and team training. Each phase hands clear context to the next, with all findings timestamped in a shared incident document for the postmortem.

It orchestrates specialized subagents (incident-responder, observability-engineer, debugger, security-auditor, performance-engineer, backend-architect, deployment-engineer, docs-architect) and references concrete tooling throughout - PagerDuty/Opsgenie, OpenTelemetry/Jaeger, Prometheus/Grafana/DataDog, ELK/Splunk, status-page tools, and Confluence/Notion for documentation.

Incident commanders, SREs, and on-call engineers running a structured, multi-phase response to a production incident - from detection through mitigation, communication, and a blameless postmortem - rather than an unstructured, single-person fire drill.