Implement Service Mesh Observability

This skill provides a complete reference for observability across Istio and Linkerd service meshes, organized around the three pillars of observability (metrics, traces, logs) and four golden signals for a mesh (latency, traffic, errors, saturation, each with a named alert threshold: P99 > 500ms, anomaly detection, >1% error rate, >80% saturation). It supplies seven ready-to-adapt templates covering the concrete stack pieces needed to instrument a mesh end to end.

Use it when setting up distributed tracing across services, implementing service mesh metrics and dashboards, debugging latency and error issues, defining SLOs for service communication, visualizing service dependencies, or troubleshooting mesh connectivity. It's scoped to service mesh observability specifically - skip it for unrelated tasks or a different domain/tool, and open resources/implementation-playbook.md when detailed examples are needed beyond what's here.

The seven templates are: (1) Istio with Prometheus & Grafana - a ConfigMap and ServiceMonitor scraping istio-telemetry; (2) key Istio PromQL queries for request rate, 5xx error rate, P99 latency, TCP connections, and request size, all keyed on destination_service_name; (3) Jaeger distributed tracing - an IstioOperator config enabling tracing (with a sampling rate note: 100% in dev, lower in prod) plus a Jaeger all-in-one Deployment exposing its UDP/Thrift/Config/UI/HTTP/gRPC/Zipkin ports; (4) a Linkerd Viz dashboard, installed and queried via the Linkerd CLI; (5) a Grafana dashboard JSON with Request Rate, Error Rate (gauge with green/yellow/red thresholds at 0/1/5%), P99 Latency, and a Service Topology node graph panel; (6) Kiali service mesh visualization, wired to Prometheus, Jaeger, and Grafana URLs; and (7) an OpenTelemetry Collector config receiving OTLP and Zipkin traces, batching them, and exporting traces to Jaeger and metrics to Prometheus, alongside an Istio Telemetry v2 resource routing 10% of traces through the otel provider. A PrometheusRule template defines three alerts: HighErrorRate (>5% 5xx for 5m), HighLatency (P99 >1000ms for 5m), and MeshCertExpiring (mesh cert expiring within 7 days).

The Linkerd Viz workflow is entirely CLI-driven:

linkerd viz install | kubectl apply -f -
linkerd viz dashboard
linkerd viz top deploy/my-app
linkerd viz routes deploy/my-app --to deploy/backend
linkerd viz tap deploy/my-app --to deploy/backend
linkerd viz edges deployment -n my-namespace

Named best practices: sample appropriately (100% dev, 1-10% prod), propagate trace-context headers consistently, set up alerts for the golden signals, correlate metrics and traces via exemplars, and use hot/cold storage tiers for retention - while avoiding over-sampling, uncontrolled label cardinality, skipping dependency dashboards, and losing track of observability cost.

Platform and SRE teams instrumenting Istio or Linkerd meshes who need ready-made Prometheus/Grafana/Jaeger/Kiali/OpenTelemetry configuration templates and alerting rules rather than building mesh observability from scratch.