Automate Monte Carlo alert triage and incident response

This skill helps design, test, and deploy an automated triage agent for Monte Carlo data-observability alerts, giving building blocks rather than a fixed workflow: a set of MCP tools, a description of each triage stage, and a working example to adapt. Tools cover fetching recent alerts, scoring incident likelihood and potential impact as HIGH, MEDIUM, or LOW, running deep root-cause analysis on high-signal alerts asynchronously with a separate poll tool keyed by incident ID, and write actions - updating alert status or severity, assigning an owner by email, posting or updating a triage comment, and marking anomaly events as normal to trigger ML threshold recalibration and stop re-alerting on the same pattern. The high-level flow is fetch alerts, score them, deep-troubleshoot high-signal ones, classify, then take actions - customizable at every stage via a referenced triage-stages guide. It recommends a maturity arc: start in recommendation-only mode where the agent posts what it would do with no real changes, to tune the workflow against how the team actually responds, then automate on a schedule while staying in recommendation mode to validate on real traffic, then finally swap recommendations for real actions once confident.

When activated it follows a strict sequence: verify the core MCP tools are accessible; determine whether the user wants interactive triage right now or to build or refine an automated workflow; for interactive triage, clarify scope, fetch and score alerts, offer deeper troubleshooting on medium-or-higher alerts with confirmation, then proactively offer write actions like status updates or marking natural variation as normal; for automated workflows, either start from the built-in example, which runs in recommendation mode with no writes on the first pass, adapt an existing workflow file, or build one from scratch iteratively. A hard action guard prevents calling any write tool while building or testing a workflow regardless of what the workflow file says, until the user explicitly switches to action mode for a production run; first runs always proceed step by step with a summary and confirmation after each stage. It closes by offering to save the workflow to the project and proposing next steps such as refining, testing on a different alert set, or scheduling via the /schedule skill.

Use it to triage recent Monte Carlo alerts interactively, or to design, refine, or deploy an automated triage workflow. Do not use it for investigating one already-known specific incident, which should be handled directly, creating or configuring monitors, which belongs to the monitoring-advisor skill, or running pre-change impact analysis, which belongs to the prevent skill.

Given alerts and a time window, it produces scored and troubleshot alert assessments and, in workflow mode, either recommended or executed actions - status updates, owner assignment, comments, incident declarations - plus an optional saved workflow file for reuse.

Monte Carlo's MCP server, routed exclusively through this plugin's bundled server via fully-qualified tool names rather than any separately-configured monte-carlo-mcp server that may point at a different endpoint; it also ties into the /schedule skill for cadence automation.

Data reliability and observability teams moving from manual Monte Carlo alert review toward semi-automated or fully automated triage, who want a safe, staged path - recommend, then validate, then act - rather than flipping straight to autonomous actions on production alerts.