Execute test plans and file F1-grade quality reports

Executes test plans and files reports using the WJTTC (F1-inspired) testing standard - the "driver," not the "engineer" that plans/generates the suite (that's the sibling skill wjttc-builder). Every test is triaged into one of five tiers by blast radius: Brake (life-critical - data loss, auth bypass, payment errors, destructive ops without confirmation), Engine (performance-critical - API accuracy, calculations, format compliance), Aero (polish/edge cases - UI quirks, docs), Tyre (durability under load - stress, concurrency, memory growth), and Pit (the release gate - smoke/regression suite, CI green, the filed report). Brake tests run first, since nothing else matters if the brakes don't work.

Before adding or running anything new, it runs a Signal Integrity pre-audit: classify the last 30 days of CI failures into Real bug (fixed by a code change), Flake (timing/network/concurrency noise, passed on rerun), or Infra (missing secret, runner change, upstream dependency), then compute SI = Real bugs / (Real bugs + Flakes + Infra) x 100. SI maps to a verdict and action: 100% maintain, 95-99% Championship (annotate flakes immediately), 85-94% Acceptable (schedule the flake fix this sprint), 70-84% Eroding (stop adding tests, fix flakes first), below 70% Dead signal (block merges until restored). It eliminates on sight: hard absolute-time perf assertions on shared runners, un-mocked network calls in the main suite, unordered concurrency tests, and secret-dependent steps that hard-fail when the secret is missing. The inverse rule also applies - a real bug that shipped despite green CI needs its regression test written before the fix lands.

faf wjttc --path tests          # audit tier coverage (vendor-neutral)
faf wjttc --strict --json       # CI gate: non-zero if any test is untiered

The execution loop is: scope (happy path, edges, failure modes, perf targets, tier of each), audit signal integrity before trusting/extending the suite, run each test (setup, execute, observe actual vs. expected, record pass/fail/blocked, capture failure evidence), reproduce and root-cause every failure deterministically, confirm every test is tiered (faf wjttc --strict --json, non-zero exit if any test is untiered), then file the WJTTC report and surface the tier verdict. Reports are YAML files saved to ./wjttc-reports/ in the project under test (never an absolute/personal path), named YYYY-MM-DD-{project}-{feature}-tests.yaml, containing a summary with pass/fail/blocked totals, per-failure root-cause and fix, edge-case results, performance measurements against targets, bugs found (tier doubles as severity), coverage (tested vs. not-tested), and a verdict.

Pass rate or SI score maps to a canonical FAF tier ladder: 100% Trophy, 99% Gold, 95% Silver, 85% Bronze, 70% Green, 55% Yellow, 1% Red, 0% White - deterministic, same input always producing the same score. Method notes: test with real (anonymized production or messy) data, not just sanitized inputs; document every failure for reproducibility; tier before testing since severity is the tier; and wire results into CI via faf taf setup --write (test receipts) plus faf score --json for a deterministic score snapshot.

Reach for this specifically when a team's CI has started feeling untrustworthy - green builds that still ship bugs, or red builds nobody investigates anymore - since the Signal Integrity pre-audit is what surfaces whether that mistrust is justified before any new test gets written or extended. It's a poor fit for a brand-new project with no test suite or CI history yet, since there's no signal to audit and no existing tests to tier; start with wjttc-builder to generate the initial suite, then bring this skill in once there's a real pass/fail history to reason about.

Input is an existing test plan/suite and, for the signal audit, 30 days of CI failure history. Output is executed test results, root-caused failures with fixes, and a YAML WJTTC report filed to ./wjttc-reports/ with a tier verdict.

Built on the faf-cli tool (faf wjttc, faf taf setup, faf score) for tier-coverage auditing and CI test-receipt wiring.

Teams running and reporting on test suites who want severity-tiered results, an honest CI signal-integrity audit, and a standardized, deterministic pass/fail report rather than an unstructured test log.