Configure Alertmanager Routing and Notifications

This skill covers Prometheus Alertmanager configuration - alert routing rules, notification management, inhibition rules, and silence configuration - with coverage of alert grouping, rate limiting, escalation patterns, and multi-channel notification integration. It documents the routing tree structure (global, route with nested routes, inhibit_rules, receivers, templates), hierarchical label matching (evaluated top-down, first match wins), grouping strategy (group_by, group_wait, group_interval, repeat_interval), and advanced matching via match, match_re, and matchers (including regex and negation on labels like severity, service, instance).

It covers alert-cascade prevention through inhibit_rules - for example a NodeDown alert inhibiting all other Node.* alerts on the same instance, critical alerts inhibiting warnings for the same alertname/service/instance, and a MaintenanceMode alert inhibiting everything on the same cluster - plus multi-channel receiver configuration (Slack slack_configs with templated title/text, PagerDuty pagerduty_configs with routing_key, email email_configs with HTML templates), time-based routing via time_intervals (business-hours vs. weekend on-call), and escalation routing that uses continue: true to fall through to an escalation receiver when a critical alert isn't resolved. It also documents validation tooling (alertmanager --config.check, amtool config routes test, amtool alert add) and performance guidance: put specific matches early in the route tree, minimize regex on hot paths, group by low-cardinality stable labels, and use continue: true sparingly.

Use it when writing or reviewing Alertmanager routing, inhibition, or receiver configuration - designing an escalation tree, wiring Slack/PagerDuty/email receivers, or debugging noisy or duplicate alerts. It is not a Prometheus alerting-rule (PromQL) authoring guide - it assumes alerts are already firing and focuses purely on how Alertmanager routes, groups, silences, and notifies on them.

Given a routing requirement - for example "critical alerts should page immediately, dev alerts should batch daily" - it produces Alertmanager YAML such as:

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'default-receiver'
  routes:
    # Critical alerts - immediate notification
    - match:
        severity: critical
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 1h
      receiver: 'critical-alerts'
    
    # Production environment alerts
    - match:
        environment: production
      group_by: ['alertname', 'instance']
      receiver: 'prod-team'
      routes:
        # Database alerts to DBA team
        - match_re:
            service: '^(mysql|postgresql|redis).*'
          receiver: 'dba-team'
        
        # Application alerts during business hours
        - match:
            team: backend
          receiver: 'backend-oncall'
          active_time_intervals:
            - business-hours
    
    # Development environment - reduced frequency
    - match:
        environment: development
      group_interval: 30m
      repeat_interval: 24h
      receiver: 'dev-team'

Configures routing to Slack (slack_configs), PagerDuty (pagerduty_configs), and email (email_configs) receivers, and validates configuration with the amtool CLI and alertmanager --config.check.

SREs and on-call engineers designing or maintaining Prometheus Alertmanager routing, escalation, and notification configuration.