Create Datadog Monitors

Datadog Monitor Creator is a skill for building comprehensive Datadog monitors for infrastructure and application monitoring, covering alerting best practices, threshold configuration, notification strategy, and how to minimize alert fatigue while still catching critical issues early.

It distinguishes four monitor types by use case: metric monitors for quantitative data like CPU, memory, request rate, and custom business metrics; APM monitors for application performance - error rates, latency percentiles, and service dependencies; log monitors for detecting specific error patterns, security events, or critical business messages in logs; and composite monitors that combine multiple conditions for scenarios needing several signals at once. For thresholds it distinguishes static thresholds (a critical tier for system-down/severely-degraded states like over 95% CPU for 10+ minutes, a warning tier for likely degradation like over 80% CPU for 5+ minutes, with different thresholds for business hours versus off-hours) from dynamic thresholds (anomaly detection for seasonally patterned metrics, bounds set from historical data such as 2-3 standard deviations, accounting for weekly and daily traffic patterns).

For advanced patterns it lays out a four-tier alert-priority strategy - P1 critical (service fully down, immediate response), P2 high (significant degradation, respond within an hour), P3 medium (minor issues, business-hours response), and P4 low (informational, no immediate action needed) - plus composite monitors that combine sub-monitors with boolean logic (for example, alerting when a high connection count AND either slow queries OR high database CPU are both true). Notification routing separates channels by purpose: PagerDuty for critical production alerts only, Slack team channels for relevant alerts, email for summary reports and non-urgent notices, and webhooks for ITSM tool integration.

Use it when you need concrete, production-ready Datadog monitor configurations - metric, APM, log, or composite - with well-calibrated thresholds and routed notifications, not just a bare metric-alert query. It explicitly warns against several anti-patterns: alert fatigue from too many low-priority alerts, "flapping" monitors that rapidly trigger and recover from badly-tuned thresholds, misconfigured no-data alerts, ignoring timezone differences across regions, alerts with no runbook or debug context, and missing recovery notifications. If your monitoring setup already suffers from these, that's the signal to apply this skill's threshold and template patterns rather than adding more raw alerts.

Input is the system or service to monitor and its expected behavior (normal ranges, seasonal patterns, business-hours schedule). Output is a Datadog monitor JSON configuration with a query, a message template with actionable links (dashboard, runbook, logs) and escalation mentions, and threshold/notification options tuned to the monitor's priority tier. A critical-infrastructure monitor sets type to metric alert with a query like avg(last_10m):avg:system.cpu.user{env:prod} by {host} > 90, critical/warning thresholds of 90/75, notify_no_data on with a 20-unit timeframe, and a 60-unit evaluation_delay. The message template mentions the relevant Slack and PagerDuty handles, states the host and current/threshold values, and links out to the host dashboard and an internal runbook URL.

Maintenance guidance covers a review cadence (monthly for alert frequency and false-positive rate, quarterly for threshold adjustment, post-incident updates from lessons learned, and temporary sensitivity changes during deploys), performance tuning (appropriate evaluation windows and delays, template variables for reuse across environments, consistent tagging), and testing (validate monitors in staging first, use scheduled downtimes during maintenance, and regularly verify notification channels).

It targets the Datadog monitor API directly across metric alert, log alert, and composite monitor types - covering examples like an API error-rate spike monitor (critical at 0.05, warning at 0.02) and a log-based alert for more than 10 failed logins in a 5-minute window - with notification routing to PagerDuty, Slack, email, and generic webhooks for ITSM integration.

SRE and DevOps engineers configuring Datadog alerting who want calibrated thresholds, priority-tiered routing, and actionable message templates - rather than noisy, under-contextualized alerts that create fatigue.