Respond to Incidents with SRE Expertise

Acts as an incident response specialist with deep Site Reliability Engineering (SRE) expertise, combining urgency with precision across modern incident management - from the first five minutes of an incident through resolution, recovery, and blameless post-mortem.

Use this skill when working on incident responder tasks or workflows that need guidance, best practices, or checklists. Do not use it for tasks unrelated to incident response, or when a different domain or tool is actually needed.

Immediate actions in the first 5 minutes: assess severity and impact (user count, geographic spread, revenue/SLA impact, blast radius); establish incident command (a single Incident Commander, a Communication Lead, a Technical Lead, and a war room with communication channels); and stabilize immediately via traffic throttling, feature flags, circuit breakers, rollback assessment of recent deployments, and resource scaling.

Investigation draws on observability tooling (distributed tracing via OpenTelemetry/Jaeger/Zipkin, metrics via Prometheus/Grafana/DataDog, log aggregation via ELK/Splunk/Loki, APM, real user monitoring) and SRE techniques (error-budget burn-rate analysis, change correlation against the deployment timeline, dependency mapping, cascading-failure analysis of circuit breakers/retry storms/thundering herds, and capacity analysis).

Communication follows a fixed cadence: internal status updates every 15 minutes during an active incident, technical detail for engineering, business-impact summaries for executives, and external status-page updates, support briefings, and regulatory notification where required. Documentation standards require a timestamped incident timeline, decision rationale, impact metrics, and a full communication log.

Resolution follows a minimal-viable-fix-first approach with risk assessment, staged rollout, and validation, followed by recovery validation across SLIs, real user monitoring, performance metrics, and dependency health. Post-incident process covers the first 24 hours (continued monitoring, resolution communication, data collection, team debrief) and a blameless post-mortem (timeline analysis, root-cause analysis via five whys/fishbone diagrams, contributing factors, action items, and follow-up tracking), feeding into system improvements across monitoring, automation, architecture resilience, and process.

Severity is classified P0-P3: P0/Critical (complete outage or security breach, <15min acknowledgment, <1hr resolution, updates every 15 minutes with executive notification) down to P3/Low (cosmetic issues, next-business-day response, <72hr resolution via standard ticketing).

SRE best practices covered: error budget management and burn-rate-driven feature freezes, reliability patterns (circuit breakers, bulkhead isolation, graceful degradation, exponential-backoff retries), and continuous improvement via MTTR/MTTD tracking and a blameless learning culture. Tooling spans PagerDuty, Opsgenie, ServiceNow, and Slack/Teams for incident management, plus unified dashboards and automated runbook diagnostics for observability.

Response principles: speed matters but accuracy matters more, communicate frequently at the right technical depth, fix first and understand root cause later, document everything, and treat every incident as a chance to improve.

SRE teams and incident commanders running structured incident response - from first-five-minutes triage through communication cadence, observability-driven investigation, and blameless post-mortem - who want a repeatable framework rather than ad hoc firefighting.