Audit a test suite

Audit a test suite measures three practical questions about a Python pytest suite: whether it notices broken code, whether it is fast and stable enough to run on every change, and which tests consume time without adding detection. A pinned runner performs the measurements; the skill interprets its reports. The workflow is report-only and does not edit, delete, skip or quarantine tests or change source code without a later request for a specific change.

Quick mode has a five-minute budget and includes one baseline run, timings, coverage and overlap, without mutants. Deep mode defaults to thirty minutes and adds three file-order runs, three random-order runs, every deletion mutant and a seeded sample of classical mutants until the budget ends. Larger projects can require a longer budget. A partial or interrupted run must be identified before presenting findings.

Use this skill when reviewing, speeding up, pruning or strengthening pytest tests, or investigating which tests are useful, slow or flaky. For checking a single change, use agentic-verify instead. The audit is limited to Python and pytest.

Interpret results conservatively. A surviving mutant identifies a candidate gap rather than proving a bug, and possible equivalent mutants need human review. Coverage overlap alone does not establish redundancy. Tests that appear redundant individually cannot be assumed redundant as a group. Inferred findings retain their inferred label.

A proposed removal set is an upper bound. Before presenting it as removable, run prune-check against held-out mutants from the audit run, with at least ten minutes of budget. Report either refuted, including lost kills or lines, or not refuted, including the killed-mutant count and resulting bound. Not refuted does not mean safe. For refuted sets, retain tests identified as sole killers and treat the remaining removals as unproven.

The first argument selects quick or deep, with quick as the default; remaining arguments go to the runner. Confirm that the target uses pytest. The runner reads test-suite-audit.toml or discovers source packages, test directories and an interpreter, checking .venv/bin/python before python3. Ask before writing configuration when discovery is unsuitable. Configuration can include protected node prefixes or markers that must never be proposed for removal.

The last stdout line is JSON containing report paths. Read report.json for evidence and report.md for the presentation. Organize findings around broken-code detection, suitability for running on every change, and bloat. Preserve labels, evidence and next validation steps. Include a reproduction command for a mutant finding, identify unassessed areas and link the report. Detection findings include weak tests, uncovered code, pseudo-tested functions and deletable calls; runtime findings include setup cost and order dependence.

The runner needs Python 3.12+ and pytest-cov; order checks need pytest-randomly. Missing coverage support must be disclosed, and installation requires permission. It runs in a disposable project copy and writes only to .test-suite-audit/. Review the runner fetched through uvx at v0.3.0 before first use, and bound execution with a Bash timeout two minutes longer than the budget.

AV='uvx --from git+https://github.com/timini/agentic-verification@v0.3.0#subdirectory=runner agentic-verification'
$AV quick . --budget-minutes 5