Create AWS CloudWatch Alarms with Best Practices

This skill covers AWS CloudWatch monitoring and alarm creation, with deep knowledge of metrics, thresholds, statistical analysis, and notification strategies designed to balance alert fatigue against operational awareness. Its core principles are basing alarm thresholds on historical data, business requirements, and operational capacity, choosing the right statistic (Average, Sum, Maximum) per metric type, balancing evaluation-period responsiveness against noise suppression, ensuring every alarm has a clear remediation path and owner, and designing cost-efficient alarm strategies. Its threshold strategy uses percentile-based thresholds (P95, P99) for latency metrics, absolute thresholds for error rate and availability metrics, tiered Warning/Critical alerts for gradual degradation, and accounts for seasonal traffic patterns. Evaluation-window guidance covers using 2-of-3 datapoints to filter transient spikes, longer evaluation periods (10-15 minutes) for autoscaling triggers, and shorter periods (1-2 minutes) for critical system failures, accounting for metric-publication delay in timing.

Use this skill when designing CloudWatch alarms for EC2, Application Load Balancer, RDS, or custom application metrics that need production-grade thresholds, evaluation windows, and notification routing rather than default settings. It covers composite alarms (combining multiple metrics with AND/OR/NOT logic for complex failure scenarios and dependency-aware alerting), CloudWatch Anomaly Detection for dynamic thresholds on cyclical or trending metrics, and all four missing-data treatment modes (notBreaching, breaching, ignore, missing) with guidance on when each applies. It is not a general AWS monitoring architecture guide - it's scoped specifically to alarm configuration, thresholds, and notification routing, not to broader observability tooling choices or dashboard design.

resource "aws_cloudwatch_metric_alarm" "ec2_cpu_high" {
  alarm_name          = "${var.instance_name}-cpu-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "3"
  datapoints_to_alarm = "2"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = "300"
  statistic           = "Average"
  threshold           = "80"
  alarm_description   = "This metric monitors ec2 cpu utilization"
  alarm_actions       = [aws_sns_topic.alerts.arn]
  ok_actions         = [aws_sns_topic.alerts.arn]
  
  dimensions = {
    InstanceId = var.instance_id
  }
  
  tags = {
    Environment = var.environment
    Team        = var.team
  }
}

Given a resource type and its failure modes, the skill produces both raw JSON alarm definitions (EC2 CPU utilization, ALB target response time with notBreaching missing-data handling, RDS connection count) and Terraform HCL resources like the one above, including custom-metric alarms built on metric_query blocks for application-level error rates. It also outputs SNS topic configurations with email and Lambda subscription endpoints for routing alerts to the right channel.

DevOps and platform engineers setting up production CloudWatch monitoring for EC2, ALB, RDS, or custom application metrics who need concrete threshold values, evaluation-period tradeoffs, and infrastructure-as-code alarm definitions rather than trial-and-error tuning. It suits teams that want cost-optimization guidance (grouping related alarms, using composite alarms instead of many individual ones, suppressing alarms during maintenance windows, tag-based consolidation) and testing discipline (using the SetAlarmState API to validate notifications, documenting alarm runbooks, and integrating with incident tools like PagerDuty or Opsgenie).