Setup Observability and Monitoring Systems

Monitoring and Observability Setup is a skill that positions the model as a monitoring and observability expert. It sets up metrics collection, distributed tracing and log aggregation, and creates dashboards that give full visibility into system health and performance. The focus is the three pillars of observability (metrics, logs, traces), the monitoring infrastructure itself, actionable dashboards and effective alerting strategies. The goal is a system that provides actionable insights, reduces MTTR and enables proactive issue detection.

The playbook has seven parts:

1. Prometheus and metrics setup: a prometheus.yml with 15 second scrape and evaluation intervals, external labels for cluster and region, Alertmanager targets, alert and recording rule files, scrape jobs for Prometheus, a node exporter and Kubernetes pods selected by annotation, and a TypeScript MetricsCollector built on prom-client. It defines a request duration histogram and a request counter labelled by method, route and status code, and an HTTP middleware that records them.
2. Grafana dashboards: a service dashboard generator with golden signal panels for request rate, error rate and latency percentiles at p50, p95 and p99, using a 6 hour window and 30 second refresh.
3. Distributed tracing: an OpenTelemetry setup for Node with a Jaeger exporter, a batch span processor, auto-instrumentation and service name, version and environment resource attributes.
4. Log aggregation: a Fluentd configuration that tails container logs, adds Kubernetes metadata and cluster and environment fields, and ships to Elasticsearch through a file buffer, plus a Python structured logging library that writes JSON entries with a timestamp, level, service, version, environment and trace context.
5. Alert configuration: rules for high error rate (critical, after 5 minutes), slow response time (warning, 10 minutes), high CPU usage (warning, 15 minutes) and high memory usage (critical, 10 minutes), with Alertmanager routing critical alerts to PagerDuty and critical or warning alerts to Slack.
6. SLO implementation: an SLO manager with a target such as 99.9 percent over a 30 day window, an error budget derived from the target, recording rules for consumed error budget, and multi-window burn rates of 14.4 over 1 hour, 6 over 6 hours and 3 over 1 day.
7. Infrastructure as code: a Terraform configuration with modules for Prometheus (with 100Gi storage and 30 day retention), Grafana with a Prometheus datasource, and Alertmanager with Slack and PagerDuty credentials passed as variables.

The Prometheus global settings look like this:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

Use it when you need to implement or improve monitoring and observability: standing up metrics collection, tracing and log aggregation, building dashboards, defining alerts and setting service level objectives.

Do not use it when the task is unrelated to monitoring and observability setup or needs a different domain or tool outside this scope. The skill also states that its output is not a substitute for environment-specific validation, testing or expert review, and that you should stop and ask if required inputs, permissions, safety boundaries or success criteria are missing.

The input is the monitoring requirement, supplied as the skill's arguments. The skill asks you to clarify goals, constraints and required inputs, apply best practices and validate outcomes. The output has eight parts:

1. Infrastructure Assessment: analysis of current monitoring capabilities.
2. Monitoring Architecture: a complete monitoring stack design.
3. Implementation Plan: a step-by-step deployment guide.
4. Metric Definitions: a comprehensive metrics catalog.
5. Dashboard Templates: ready-to-use Grafana dashboards.
6. Alert Runbooks: detailed alert response procedures.
7. SLO Definitions: service level objectives and error budgets.
8. Integration Guide: service instrumentation instructions.

Platform, DevOps and site reliability engineers who need to build or improve a monitoring stack covering metrics, logs, traces, alerts and SLOs.