Debug Systematically to Find Root Causes

Systematic Debugging enforces an Iron Law - no fixes without root cause investigation first - across any technical issue: test failures, production bugs, unexpected behavior, performance problems, build failures, or integration issues. It's meant to be followed especially under time pressure or when a quick fix seems obvious, since the source's own data shows systematic debugging resolves issues in 15-30 minutes with a 95% first-time fix rate and near-zero new bugs introduced, versus 2-3 hours of thrashing and a 40% first-time fix rate for random fixes.

Four phases must complete in strict order. Phase 1 (Root Cause Investigation) requires reading error messages and full stack traces carefully, reproducing the issue consistently (gathering more data rather than guessing if it isn't reproducible), checking recent changes via git diff, and - for multi-component systems - adding diagnostic instrumentation at every component boundary before proposing any fix:

   # Layer 1: Workflow
   echo "=== Secrets available in workflow: ==="
   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

   # Layer 2: Build script
   echo "=== Env vars in build script: ==="
   env | grep IDENTITY || echo "IDENTITY not in environment"

   # Layer 3: Signing script
   echo "=== Keychain state: ==="
   security list-keychains
   security find-identity -v

   # Layer 4: Actual signing
   codesign --sign "$IDENTITY" --verbose=4 "$APP"

Phase 2 (Pattern Analysis) finds working examples in the same codebase, reads any reference implementation completely rather than skimming, and lists every difference between the working and broken cases without dismissing small ones. Phase 3 (Hypothesis and Testing) requires stating a single, specific hypothesis ("I think X is the root cause because Y"), testing it with the smallest possible change, and forming a new hypothesis rather than layering more fixes on top if it doesn't work. Phase 4 (Implementation) requires a failing test case before fixing (using superpowers:test-driven-development), a single targeted fix with no bundled improvements, and verification - critically, if 3 or more fix attempts fail, the process requires stopping to question whether the architecture itself is wrong rather than attempting a fourth fix.

A table of common rationalizations is called out explicitly ('issue is simple, don't need process,' 'multiple fixes at once saves time,' 'one more fix attempt' after 2+ failures) alongside their reality checks, and a separate set of redirection phrases from a human partner - 'Stop guessing,' 'Ultrathink this,' 'We're stuck?' - are flagged as signals to immediately stop and return to Phase 1. If systematic investigation genuinely reveals no root cause (truly environmental, timing-dependent, or external), the process still requires documenting what was investigated, implementing appropriate handling (retry, timeout, error message), and adding monitoring - but the source is explicit that 95% of 'no root cause' conclusions actually reflect incomplete investigation. Supporting techniques live alongside this skill: root-cause-tracing.md for backward call-stack tracing, defense-in-depth.md for post-fix validation layering, and condition-based-waiting.md for replacing arbitrary timeouts with condition polling - plus superpowers:verification-before-completion for confirming a fix actually worked before claiming success.

Developers debugging any technical issue who need a disciplined, phase-gated process resisting the temptation to guess-and-check - especially valuable under time pressure, where the systematic approach is empirically faster than thrashing, not slower.