Debug Systematically to Find Root Causes

Systematic Debugging organizes failure analysis into root-cause investigation, pattern analysis, hypothesis testing, and implementation. Its central rule is to investigate before proposing a fix. Authorized rollback or containment can restore service during an incident, but that action is recorded as a temporary mitigation while evidence is preserved for a verified repair.

Investigation begins with the observable failure and the smallest useful evidence check. Read complete errors and stack traces, record paths and error codes, reproduce exact steps, and inspect recent commits, dependency changes, configuration, and environment differences. If reproduction is inconsistent, gather more data rather than guessing.

For multi-component failures, inspect each boundary using allowed field names, sizes, statuses, and correlation identifiers. Redact secrets and private payloads, verify configuration propagation, and compare state across layers. The goal is to locate the failing component before changing it. Trace a bad value backward through callers until its origin is understood, then repair that origin instead of masking a downstream symptom.

Pattern analysis finds comparable working code, reads the reference implementation completely, lists differences, and checks dependencies and assumptions. Hypothesis testing states one specific explanation supported by evidence and changes one variable at a time. A failed test leads to a new hypothesis rather than another fix layered onto the first.

Implementation starts with a failing reproduction or focused behavioral regression. Make one root-cause change, confirm that the original issue disappears, and check that other tests remain sound. Unrelated refactoring and opportunistic improvements are outside that repair.

Use this process for test, build, production, integration, performance, and unexpected-behavior failures. Time pressure, apparently simple issues, previous unsuccessful fixes, and incomplete understanding strengthen the case for evidence gathering.

Stop when proposing fixes before tracing the failure, combining several changes, skipping the reproduction, or adapting an incompletely understood reference. Human feedback that challenges an assumption or asks for better evidence is another reason to return to investigation.

After three unsuccessful fixes, discuss architecture and coupling with the human partner before another attempt. Shared-state problems, fixes requiring extensive refactoring, and new symptoms elsewhere are reasons to reassess assumptions; repeated failures alone do not prove that the architecture is wrong.

If completed investigation indicates an environmental, timing-dependent, or external issue, document the work, add suitable retry, timeout, or error handling, and monitor the remaining uncertainty. State what evidence would distinguish that explanation from an implementation defect.

Inputs include a reproducible command or observed failure, exact revision and runtime, recent changes, and an authorized test environment. Outputs include boundary evidence, a traced cause or explicitly unresolved explanation, a tested hypothesis, a regression check, and verification records. Keep configuration-presence checks separate from read-only artifact validation:

codesign --verify --verbose=2 "$APP"

Successful artifact verification does not establish that the intended signing identity propagated. For a build that succeeds locally but fails in CI, check identity presence at each subprocess boundary without logging its value, fix forwarding once, rerun the failing build, and validate the artifact separately.

Supporting material covers backward tracing, layered validation after identifying the cause, and condition-based waiting. Captured logs require inspection before sharing. The polluter helper executes project tests and belongs in an isolated fixture or checkout with a compatible runner; waiting examples need domain adapters and verification of the actual timeout path. Historical case counts are illustrations rather than current measurements.