Configure Prometheus for Monitoring

Prometheus Configuration is a complete guide to Prometheus setup, metric collection, scrape configuration, and recording rules, architected as instrumented applications exposing a /metrics endpoint that Prometheus scrapes periodically, feeding AlertManager for alerts, Grafana for visualization, and Thanos/Cortex for long-term storage. Installation covers Kubernetes via the kube-prometheus-stack Helm chart (with retention and storage size flags) and Docker Compose. The core prometheus.yml configuration sets global scrape/evaluation intervals and external labels, wires up Alertmanager, loads rule files, and defines scrape jobs - Prometheus self-monitoring, static node-exporter targets with relabeling, Kubernetes pod discovery filtered by prometheus.io annotations, and an application job with mTLS (ca_file/cert_file/key_file). Scrape configuration patterns cover static targets with labels, file-based service discovery reading JSON/YAML target files on a refresh interval, and Kubernetes service discovery filtered and relabeled from service annotations. Recording rules pre-compute expensive queries - HTTP request rate, error rate percentage, and P95 latency via histogram_quantile for API metrics, plus CPU/memory/disk utilization percentages for resource metrics - each on its own evaluation interval. Alert rules use those recording rules as thresholds:

- alert: ServiceDown
        expr: up{job="my-app"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.instance }} is down"
          description: "{{ $labels.job }} has been down for more than 1 minute"

alongside HighErrorRate, HighLatency, HighCPUUsage, HighMemoryUsage, and DiskSpaceLow alerts, each with a for duration, severity label, and templated summary/description.

Use this skill when setting up Prometheus monitoring, configuring metric scraping, creating recording rules, designing alert rules, or implementing service discovery.

Given a Prometheus task, the skill outputs the matching YAML configuration - scrape job, recording rule, or alert rule - plus validation commands (promtool check config, promtool check rules, promtool query instant) and troubleshooting API calls to inspect live scrape targets, running configuration, and ad-hoc queries via curl against the Prometheus HTTP API.

Points to four reference/asset files - assets/prometheus.yml.template, references/scrape-configs.md, references/recording-rules.md, and scripts/validate-prometheus.sh - and three related skills: grafana-dashboards for visualization, slo-implementation for SLO monitoring, and distributed-tracing for request tracing. Ten best practices are stated: use consistent metric naming (prefix_name_unit), set scrape intervals of 15-60s, use recording rules for expensive queries, run multiple Prometheus instances for high availability, size retention to storage capacity, use relabeling for metric cleanup, monitor Prometheus itself, implement federation for large deployments, use Thanos or Cortex for long-term storage, and document custom metrics.

Platform and observability engineers setting up or extending Prometheus monitoring who need working scrape configurations across static, file-based, and Kubernetes service discovery, plus recording and alert rules already wired to real metric expressions, rather than writing PromQL and relabeling rules from scratch.