Diagnose data pipeline performance bottlenecks

A Monte Carlo skill that diagnoses data pipeline performance issues - bottlenecks, regressions, and root causes - across Airflow, dbt, Databricks, and warehouse query engines, using 9 tools organized into a tiered investigation. Tier 1 discovery calls get_jobs_performance (slow/failing jobs, optionally filtered by integration type, returning average duration, 7-day trend, run count, and failure rate) and get_top_slow_queries (slowest query groups by total runtime, filterable to read or write queries). A bridge tool, get_tables_for_job, converts a problematic job's MCON to table MCONs for deeper investigation. Tier 2 diagnosis then drills in: get_tasks_performance finds the bottleneck task within a job; get_change_timeline - described as the most powerful tool - returns a single unified timeline of query text changes, volume shifts, and Airflow/dbt task failures, surfacing correlations like a query change followed by doubled runtime the next day; get_query_rca gives pre-computed root-cause groupings for failed queries (errors, timeouts, permissions) and futile queries (ones that run but produce no useful output); get_query_latency_distribution shows p50-vs-p95 trends to distinguish outlier-driven latency from a genuine regression, with an hourly bucket option since the default daily downsampling hides hour-level step-changes; and get_asset_lineage traces downstream impact or upstream causes. A supporting get_warehouses call lists available warehouses to scope the investigation. The skill enforces quoting tool-returned numbers exactly rather than rounding or estimating, comparing against 7-day trend baselines to separate regressions from normal variance, and stopping once a root cause is found - a typical investigation takes 3-7 tool calls, and more than 10 signals over-investigation.

Use it when someone asks why a pipeline, job, or query is slow, wants to find expensive or costly queries, mentions a performance regression, or wants to compare performance over time or find bottleneck tasks. Do NOT use it for data quality issues (the prevent skill), storage cost questions (the storage-cost-analysis skill), creating monitors (the monitoring-advisor skill), or just querying or exploring table contents.

Input is a performance question - a specific job, pipeline, or table, or a general "what's slow" discovery request - optionally scoped to a platform (Airflow, dbt, Databricks) or query type (read or write). Output is a structured finding: a problem summary with exact numbers, a root cause (what changed), the downstream impact, and specific recommended actions.

Calls Monte Carlo's cross-platform observability tools exclusively through the plugin's bundled MCP server using fully-qualified tool names, never a separately-configured monte-carlo-mcp server in the same session, and works across Airflow, dbt, Databricks, and warehouse query engines without needing separate platform-specific tooling.

Data engineers troubleshooting slow pipelines, expensive queries, or performance regressions who want a structured, numbers-quoted root-cause investigation across their orchestration and warehouse stack rather than manually correlating job logs, query history, and lineage by hand.