Generate Production-Ready Prometheus Configurations

This skill acts as a Prometheus configuration expert, generating comprehensive, production-ready monitoring configurations: global settings (scrape_interval, evaluation_interval, scrape_timeout, external_labels), scrape configs for static targets and service discovery (Consul, Kubernetes), alerting rules, recording rules, and relabeling patterns. It applies best practices such as setting scrape_timeout below scrape_interval, using job names that clearly identify the service, implementing proper authentication and TLS, and using recording rules for frequently queried complex expressions.

It produces Kubernetes service-discovery configs for API servers and annotated pods (via kubernetes_sd_configs and relabel_configs), essential alerting rules covering instance-down, high-CPU, high-memory and low-disk-space conditions, recording rules for CPU/memory/filesystem utilization, advanced relabeling patterns for dropping high-cardinality metrics and renaming labels, and secure scrape configs with TLS client certificates and basic auth.

groups:
- name: infrastructure.rules
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

  - alert: HighCPUUsage
    expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage on {{ $labels.instance }}"
      description: "CPU usage is above 80% for more than 10 minutes."

Use it when standing up or hardening a Prometheus deployment and you need scrape configs, alerting rules, recording rules, or service-discovery setups (static, Consul, Kubernetes) that follow established best practices around intervals, relabeling, security, and cardinality control. It is well suited to production hardening: adding TLS/basic-auth to scrape targets, setting up Kubernetes pod/API-server discovery, or writing alert rules for instance-down, CPU, memory and disk conditions.

It is not a Prometheus deployment or installation tool, and it does not manage Alertmanager routing or notification channels beyond referencing an alertmanagers target list - it generates the YAML configuration content itself.

Input is a description of what needs to be monitored - services, Kubernetes workloads, infrastructure hosts - and any specific requirements (retention, federation, security). Output is Prometheus-compatible YAML: global config blocks, scrape_configs (static_configs, consul_sd_configs, kubernetes_sd_configs), alerting rule groups, recording rule groups, and relabel_configs, following the naming, interval, and security conventions described in the skill.

Covers service discovery integration with Consul (consul_sd_configs) and Kubernetes (kubernetes_sd_configs for endpoints and pod roles, using annotation-based scrape configuration), Alertmanager (alerting.alertmanagers block with API v2), and exporters such as Node Exporter for system metrics.

DevOps and SRE engineers configuring or hardening Prometheus monitoring - writing scrape configs, Kubernetes service discovery, alerting rules, and recording rules - who want configurations that already follow interval, security, and cardinality best practices rather than starting from a blank YAML file.