Instrument Production Code with Observability Telemetry

A 7-step observability and instrumentation process built on the premise that instrumentation belongs alongside the feature, not after launch. Step 1 forces writing down 2-4 concrete questions an on-call engineer will ask about the feature before adding any telemetry - if you can't name the questions, you're not ready to instrument. Step 2 picks the right signal per question: structured logs answer what happened in a specific case, metrics answer how often or how fast in aggregate, traces answer where time went across services - metrics tell you that something is wrong, traces tell you where, logs tell you why. Step 3 covers structured logging: every line is a JSON object with a stable event name and machine-readable fields (never string interpolation), a four-level severity scale (error/warn/info/debug) mapped to specific on-call actions, mandatory correlation/request IDs generated at the system boundary and propagated to every log line and span, and a hard rule against logging secrets, tokens, or full PII. Step 4 covers metrics: RED (Rate, Errors, Duration) for request-driven services and USE (Utilization, Saturation, Errors) for resources, via the vendor-neutral OpenTelemetry metrics API (Prometheus' prom-client shown as one example backend), with cardinality as the named failure mode - labels must come from small fixed sets like route templates or status classes, never user IDs or raw URLs - and latency tracked as histograms with p50/p95/p99, never a plain average. Step 5 covers distributed tracing via OpenTelemetry's auto-instrumentation for HTTP/gRPC/DB clients, manual spans only around meaningful internal work, context propagated across every async boundary or the trace breaks, and head-based sampling at a low rate with 100% of errors kept if the backend supports tail sampling. Step 6 covers alerting: alert on symptoms users feel (error rate, p99 latency, queue age) rather than causes (CPU, one pod restart), with four rules for every alert - it must be actionable, it links to a runbook, its threshold is justified by an SLO or historical data rather than a guess, and only two severities exist (page for user-facing issues needing action now, ticket for degradation this week), since a third tier becomes noise. Step 7 verifies the telemetry itself before calling the work done - forcing a staging error and finding it in logs by request ID, sending test traffic to confirm metric labels and values, following one request across services with no broken spans, and test-firing every new alert to confirm it reaches the right channel with a working runbook link.

Use it when building any feature that will run in production, adding a service, endpoint, background job, or external integration, after a production incident took too long to diagnose, when setting up or reviewing alerting rules, or reviewing a PR that adds I/O, retries, queues, or cross-service calls. Do NOT use it for diagnosing a failure happening right now (that's the debugging-and-error-recovery skill, which this instrumentation is what makes fast next time), profiling and optimizing already-measured slowness (the performance-optimization skill), or launch-day monitoring checklists and rollback triggers (the shipping-and-launch skill, which this instrumentation feeds).

Input is a feature or PR that will run in production, plus the on-call questions it should answer. Output is structured, correlation-ID'd logging; RED/USE metrics with bounded label sets and percentile latency; distributed traces with propagated context; and symptom-based, runbook-linked alerts - each verified by triggering the actual path and inspecting the telemetry it produces.

Uses the OpenTelemetry API as the vendor-neutral standard for both metrics and tracing (with Prometheus' prom-client shown as one example metrics backend), plus a structured JSON logger with request-scoped child loggers for correlation IDs.

Developers and reviewers who want production features to be diagnosable from telemetry alone - answering an on-call engineer's questions from logs, metrics, and traces rather than guesswork - and who want a concrete process and verification checklist rather than a vague "add some logging" instruction.