Implement Service Mesh Observability

Covers observability patterns for Istio, Linkerd and service-mesh deployments across three pillars, metrics (request rate, error rate, latency P50, saturation), traces (span context, latency, dependencies, bottlenecks) and logs (access logs, error details, debug info, audit trail), plus four golden signals for the mesh: latency (P50/P99, alert above P99 > 500ms), traffic (requests per second, anomaly detection), errors (5xx rate, alert above 1%), and saturation (resource utilization, alert above 80%).

It ships seven ready-to-apply templates: Istio with Prometheus and Grafana (a Prometheus ConfigMap scraping istio-telemetry via kubernetes_sd_configs, plus a ServiceMonitor on a 15s interval); key Istio PromQL queries for request rate, 5xx error rate, P99 latency via histogram_quantile, TCP connections and request size, all built on istio_requests_total and istio_request_duration_milliseconds_bucket; Jaeger distributed tracing (an IstioOperator config enabling tracing at 100% sampling in dev, plus a jaeger-all-in-one deployment with its UDP/Thrift/HTTP/gRPC/Zipkin ports); a Linkerd Viz dashboard installed and driven from the CLI (top, routes, tap, edges); a Grafana dashboard JSON with request-rate, error-rate gauge, P99-latency and service-topology node-graph panels; a Kiali installation wired to Prometheus, Jaeger and Grafana URLs; and an OpenTelemetry Collector config receiving OTLP/Zipkin, batching, and exporting to Jaeger and Prometheus, alongside an Istio Telemetry v2 resource routing 10% of traces through the otel provider.

A PrometheusRule alerting set defines three alerts: HighErrorRate when the 5xx ratio exceeds 5% for 5 minutes at critical severity, HighLatency when P99 exceeds 1000ms for 5 minutes at warning severity, and MeshCertExpiring when a mesh certificate has under 7 days left, at warning severity.

Best practices split into do's, sample appropriately (100% in dev, 1-10% in prod), propagate trace-context headers consistently, set up alerts for the golden signals, correlate metrics and traces via exemplars, and use hot/cold storage retention tiers, and don'ts, don't over-sample given storage cost, don't ignore label cardinality, don't skip dependency-visualizing dashboards, and don't forget to monitor observability costs itself.

Use it to set up distributed tracing across services, implement mesh metrics and dashboards, debug latency and error issues, define SLOs for service communication, visualize service dependencies, or troubleshoot mesh connectivity, on Istio or Linkerd. Skip it for anything unrelated to service-mesh observability or outside this domain.

Input is the mesh, Istio or Linkerd, and its namespace and service topology. Output is applied Kubernetes manifests, ConfigMaps, ServiceMonitors, Deployments, PrometheusRules, for Prometheus, Grafana, Jaeger, Kiali and OpenTelemetry, plus PromQL queries and CLI commands for day-to-day observability.

Integrates Prometheus, Grafana, Jaeger, Kiali and OpenTelemetry with Istio or Linkerd, exposed via kubectl apply-able manifests, PromQL queries against istio_requests_total/istio_request_duration_milliseconds_bucket, and Linkerd's own CLI:

linkerd viz install | kubectl apply -f -
linkerd viz dashboard

Platform and SRE engineers running Istio or Linkerd who need ready-made metrics, tracing, dashboard and alerting configuration for the mesh rather than building it from scratch.