Debug and Resolve System Errors

Error Analysis and Resolution is a skill for systematic root-cause analysis of errors and production incidents, from local development through distributed-systems observability. It is a compatibility alias of error-diagnostics-error-analysis, kept so existing manifests referencing this callable ID keep working; new references should use the other ID.

Its core loop: gather error context, timestamps and affected services; reproduce or narrow the issue with targeted experiments; identify the root cause and validate it with evidence; then propose fixes, tests and preventive measures. The bundled resources/implementation-playbook.md expands each step with explicit anti-shortcuts:

• Establishing the symptom means recording the timestamp/window/timezone, release revision, route or job, expected versus observed behavior, and affected population, distinguishing rate from count. A severity is chosen from actual user impact, not the exception class alone, and an error spike after a deploy is treated as a hypothesis about causality, not proof - a timeout may come from a pool, lock, network, upstream delay or cancellation, and a single-thread ConcurrentModificationException does not by itself prove a cross-thread race.
• Narrowing one hypothesis: reproduce with the smallest synthetic input in an authorized environment, follow the failing call path including adapters and retries, compare against a passing input or known-good revision, change one controlled variable at a time, and record what each experiment rules out. Five-whys can organize questions, but the fifth answer is not automatically a root cause.
• Collecting diagnostics without collecting everything: use allowlisted event fields (time, service/revision, operation, bounded error class, duration, correlation id) and explicitly avoid request bodies, tokens, session identifiers, payment details or full exceptions, which can carry private data; a small header-name blacklist is not sufficient on its own.
• A local timing helper that separates the business result from telemetry delivery, logging only status, duration and exception type, and never converting a failed telemetry sink into a business retry:

import time

def observe_call(operation, record):
    started = time.monotonic()
    event = {"status": "ok"}
    try:
        return operation()
    except Exception as error:
        event = {"status": "error", "error_type": type(error).__name__}
        raise
    finally:
        event["duration_seconds"] = time.monotonic() - started
        try:
            record(event)
        except Exception:
            # The application's telemetry health channel should report sink failures.
            # Do not replay the original business operation because logging failed.
            pass

• Retry and circuit-breaker requirements: use the established client library rather than a handwritten universal retry wrapper, define retryable failures, a total deadline, an attempt limit and jitter, never retry authentication or validation failures, and require a stable operation ID plus provider idempotency for writes since a timeout can occur after the remote side effect already completed.
• Vendor integration guidance for Sentry, Datadog or OpenTelemetry: verify initialization order, sampling, exporter destination and shutdown/flush behavior, test a synthetic error with no private data before enabling the same config for real traffic, and never send raw exception messages as span status or customer identifiers as unbounded metric labels.
• Production diagnostics rules: use the incident owner's authorized process, since a heap dump can expose all in-memory secrets and pause or exhaust the process, and traffic replay can disclose data or repeat writes even in staging.
• Alerts and communication rules: define an actionable signal with population, window, threshold rationale and owner, keep units consistent (0.05 is 5%, not 0.05%), and never fabricate named responders, metrics or resolution times in an incident update.

Use it for investigating production incidents or recurring errors, performing root-cause analysis across services, or designing observability and error-handling improvements. Do not use it for purely feature-development tasks, when you cannot access error reports, logs or traces, or for an issue unrelated to system reliability. Its safety notes: avoid production changes without approval and rollback plans, and redact secrets and PII from any shared diagnostics.

Input is $ARGUMENTS: specific error messages, stack traces, log files, failing services or general error patterns. Output is a report of evidence, the reproduced trigger, the verified root cause or remaining hypotheses, and scoped corrective action - never a diagnosis established merely from recognizing an error-string pattern.

It names integration paths for Sentry, Datadog and OpenTelemetry, and for HTTP/OTLP export to an approved encrypted, authenticated endpoint.

It suits engineers debugging distributed systems and production incidents who need a disciplined, evidence-based process rather than pattern-matching an error string to a fix.