Configure Uptime Monitoring Systems

This skill is expert in uptime monitoring systems, health check configuration, and service reliability monitoring, with deep knowledge of monitoring tools, alerting strategies, and outage-detection best practices. Its multi-layer monitoring strategy spans synthetic monitoring (external probes simulating user behavior), Real User Monitoring, infrastructure monitoring, application monitoring, and business logic monitoring for critical transactions. Its health check design principles call for both shallow and deep checks, ensuring checks don't degrade performance, including dependency validation in deep checks, and returning structured, actionable health data. It provides service-type-specific check interval configurations (web frontend 30s, API 15s, database 60s, batch jobs 300s, each with matched timeout and retry counts), a Flask health check implementation with basic and detailed endpoints checking database and Redis connectivity plus CPU/memory/disk resource thresholds, an UptimeRobot API client for creating HTTP and keyword monitors, and a full Prometheus scrape configuration including a blackbox exporter probe for external HTTP checks.

Use this skill when setting up or hardening uptime monitoring that needs real alerting discipline, not just a dashboard nobody watches. It defines concrete Prometheus alert rules - ServiceDown firing after 1 minute of down status, HighResponseTime firing when probe duration exceeds 5 seconds for 2 minutes, HighErrorRate firing when the 5xx rate exceeds 10% over 5 minutes - each with severity labels and templated annotations. It covers advanced patterns like a circuit breaker that tracks CLOSED/OPEN/HALF_OPEN state and exposes its own health status, multi-region Prometheus deployment with per-region external labels feeding a shared Grafana, and a multi-channel AlertManager routing info/warning/critical/emergency severities to Slack, email, PagerDuty, and webhook channels respectively. It is not meant for naive fixed-interval polling with no alert discipline - its smart alerting guidance explicitly calls for fatigue prevention, alert grouping/deduplication, escalation policies, and suppression during maintenance windows.

### alert_rules.yml
groups:
- name: uptime_alerts
  rules:
  - alert: ServiceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Service {{ $labels.instance }} is down"
      description: "{{ $labels.instance }} has been down for more than 1 minute"

  - alert: HighErrorRate
    expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate on {{ $labels.instance }}"

Given a service topology, the skill produces per-service-type check interval configs, Flask health check endpoints checking dependencies and system resources, UptimeRobot API automation for monitor creation, a full Prometheus scrape and alerting configuration with the rules shown above, a circuit breaker class reporting its own health state, a Docker Compose multi-region Prometheus/Grafana setup, and a severity-routed multi-channel AlertManager class.

SRE and platform engineers setting up or hardening uptime monitoring who need real alerting discipline - thresholds, severity routing, and escalation - not just a status dashboard. It suits teams running multi-region infrastructure that need distributed Prometheus deployment, and that treat monitoring reliability itself as a concern ("monitor the monitors"), version-controlling monitoring configuration as code and regularly testing alert channels.