Evaluate cheaper LLM replacements with statistical rigor

oss-migration-eval

Decide - rigorously - whether to move an expensive LLM pipeline to a cheaper or open-source model. A Claude Code skill that estimates cost savings vs accuracy loss, finds the most probable replacement models, and produces an honest go/no-go verdict with error bars.

The uncomfortable truth this skill is built around: most eval "winners" are single-sample noise. A real experiment crowned a decisive 9.00/10 winner off one generation per cell; a 5-seed rigor pass dropped it to 7.00 ± 2.0 - mid-pack in a six-way statistical tie. Within-model variance across seeds nearly equalled the entire between-model ranking. So the honest output is often "no clear winner - here's the tradeoff, and here are the models you can safely rule out." Saying that is a success, not a failure.

What it does

Given a recurring, expensive LLM task (lead scoring, extraction, classification, outbound copy, …), it:

1. Finds the fattest tasks worth migrating (pairs with token-audit).
2. Frames the accuracy metric - deterministic for structured output, a blind panel for subjective.
3. Builds a human-validated golden set on your frontier model, including hard and false-trigger trap cases.
4. Shortlists candidates live from quality (Artificial Analysis) and usage (OpenRouter) leaderboards - never a hardcoded list.
5. Sweeps each candidate provider-pinned across multiple seeds, recording measured cost + latency.
6. Grades with error bars and reports statistically tied vs clearly worse - you can only reliably name the losers, not a winner.
7. Optionally optimizes the gap with a prompt-tuning loop (train/test/val discipline; pairs with autoresearch).
8. Deploys via orchestration - the smart model orchestrates, the cheap model does the narrow task; local via Ollama; fine-tuning as a last resort.

Why it's different - the honesty guardrails

Baked in, because a naive eval lies to you:

• ≥ 3 seeds, default 5 - single-seed rankings are noise; every number ships with a 95% CI.
• "Tied" means not separable at this n, not equivalent - it prints the minimum difference detectable at your sample size.
• Case-clustered statistics - at temperature 0 the per-seed repeats are near-replicates, so effective n = cases, not cells.
• Provider-pinned to highest precision - unpinned OpenRouter routing silently serves fp4/int4 vs fp8/bf16 call-to-call; a benchmark lottery.
• Deterministic checks beat the LLM judge - a "is this number actually in the input?" check disagreed with the judges' fabrication flags, and was right.
• Mandatory hard + trap cases - easy-only golden sets over-claim "solved."
• Human-validated oracle, enforced - the scripts refuse to grade against a reference no human signed off on.
• Refuses bad-fit candidates - one-off tasks, no checkable output, safety-critical quality bars, or a frontier baseline that isn't itself stable.

Install

Clone into your Claude Code skills directory:

git clone https://github.com/BayramAnnakov/oss-migration-eval.git ~/.claude/skills/oss-migration-eval

Then just ask Claude Code things like "can a cheaper model replace my lead-scoring pipeline?", "am I overpaying for this LLM task?", or "evaluate OSS alternatives for X."

Quick start (manual harness run)

The templates/ are scaffolds you copy into a working directory and adapt:

mkdir eval && cd eval && cp ~/.claude/skills/oss-migration-eval/templates/* .
# 1. write tasks.py (schema documented at the top of build_golden.py)
export OR_KEY=<your-openrouter-key>
python3 build_golden.py        # propose references on the frontier model -> STOP, a human validates them
python3 pick_candidates.py discover   # live open-weight models + pricing
python3 pick_candidates.py pin        # highest-precision provider pins
python3 run_sweep.py           # N-seed, provider-pinned, resumable sweep
python3 grade.py               # deterministic metrics + cost/accuracy tradeoff table (no API calls)
python3 judge.py               # blind multi-family panel for subjective tasks
# then fill templates/report_template.md

A full reference run (9 models × 8 cases × 5 seeds + a 3-judge panel) costs about $1 on OpenRouter. Never cut seeds to save cents.

Requirements

• Claude Code
• Python 3.9+ (standard library only - no dependencies)
• An OpenRouter API key with a few dollars of credit

templates/report_template.md

Migration decision report - <pipeline / task name>

Date: · Incumbent: · Task volume: <N calls/month, $X/month> Harness: x x = cells, provider-pinned · eval cost $ Golden refs validated by: <name - human sign-off, or "frontier-proposed, UNVALIDATED -> verdict is provisional">

Verdict (one paragraph, up front)

<Swap / swap-with-guard / optimize-first / keep frontier - per task. One sentence each on why, with the projected annual savings at stated volume. If the top is a statistical tie, SAY SO here: "no single winner; pick on cost/speed/failure-rate among the tied cluster.">

The tradeoff table

Quality source: <blind 3-judge panel (families: X, Y, Z) / deterministic pass-rate, CASE-clustered n=>. Cost is the mean of API-reported cost across all cells - includes reasoning-token burn, not the price sheet; carry its stderr because a single runaway cell can dominate the mean. State the minimum difference detectable at this n (grade.py prints it) so "tied" is not misread as "equivalent".

Statistical reading - tied vs worse

• Top cluster (statistically tied, |z| <= 1.96): . Overlapping CIs - crowning a winner among these would be reading noise. Choose among them on cost, latency, and the deterministic failure rates below.
• Clearly worse: (z = vs top).
• Within-model spread across seeds: +/- <x.xx>; between-model range: <x.xx>. <If comparable: "per-model noise is on the order of the whole ranking - only the losers separate.">

Deterministic findings (these carry more weight than the panel)

• Fabrication (metric-like numbers absent from the input), per model across samples: <...>
• Trap / boundary cases: <N/M models fell for the false-trigger traps; boundary-case dispersion summary. Name what the traps do NOT test.>
• Format / guard violations, empty outputs: <...>

Recommendation per task

<For optimize-first: what the autoresearch loop should target and the val-gate threshold. For keep frontier: which cheaper levers (caching, prompt trim, batch tier) still apply.>

Deployment notes

<Orchestration: smart model dispatches this task to the pinned cheap model - never the whole agent. Guards wired into production. Rollback = one config flag. Re-run triggers: prompt change, pin/quant change, notable new model, or quarterly.>

Limitations (mandatory - do not trim)

This verdict is based on cases x seeds per model ( panel scores/model), on THESE prompts, THESE pinned providers, as of . It is not a general model ranking. Known blind spots: <e.g. traps test false positives only; subjective panel n gives ~+/-x.x CI; 0/N failure rates are indistinguishable from a true y% rate at this sample size; val set was consulted once/never/more-than-once>. Prices and the model field turn over monthly - treat as perishable.

references/rigor.md

Rigor - the honesty guardrails

These rules exist because a real experiment violated them, published-almost, red-teamed itself, and watched its headline reverse. Round 2 of that experiment declared a decisive winner: 9.00/10, swept every case, "clean generational win." Round 3 - same models, same cases, plus seeds, provider pins, and hard cases - dropped that winner to 7.00 +/- 2.0, mid-pack in a six-way statistical tie. Every rule below is a scar.

1. Single-seed LLM evals are noise

The killer stat: within-model spread across seeds/judges was +/- 1.92 points, while the ENTIRE between-model range (best to worst of 9 models) was 2.88 points. Per-model noise nearly equals the whole ranking. One sample per cell produced a false "9.00/10, swept all cases."

• Mandate >= 3 seeds per model x case; default 5.
• Every reported number carries mean +/- stderr (and 95% CI in the report).
• If budget forces a choice between more models and more seeds, cut models. Five models at 5 seeds beats ten models at 1 seed - the latter is a random-number generator with a bill.

2. You can identify the losers, not a winner

With overlapping CIs, "who's best" is unanswerable - but "who's clearly worse" usually is. In the reference run the top 6 were statistically indistinguishable while the bottom 2 separated cleanly (significantly below the pack).

• Compute pairwise z = (mean1 - mean2) / sqrt(se1^2 + se2^2); |z| > 1.96 separates at 95%.
• Report three buckets: top cluster (statistically tied), mid, clearly worse.
• Never crown a winner off overlapping CIs. When the top is tied, say "tied on quality - pick on cost, speed, and deterministic failure rates" - that IS the finding, and it's a more useful one than a fake winner.
• "Tied" = NOT SEPARABLE at this n = absence of evidence, NOT proof of equivalence. At small n, only huge gaps are detectable, so everything short of catastrophic "ties." Always report the minimum difference detectable at your n (grade.py prints it). If that number is bigger than a gap you'd act on, your eval is underpowered - the fix is more cases, and until then you cannot say "model A = model B."
• Aggregate at the CASE level, not the cell level. At temperature 0 the per-seed repeats are near-replicates (some providers ignore the seed param entirely - the frontier especially), so N seeds x M cases is NOT N*M independent trials. Effective n = M cases; case difficulty is heterogeneous and usually one or two hard cases carry all the separation. Treating cells as iid Bernoulli trials inflates confidence and shrinks CIs artificially. grade.py clusters by case for exactly this reason.

3. Provider/quant routing silently corrupts results

OpenRouter load-balances each model across backends serving different quantizations. The reference experiment discovered its Round 2 data had silently served one flagship at int4/fp4 - it was benchmarking a lobotomized copy and attributing the result to the model.

• Pin every candidate to ONE highest-precision provider: "provider": {"order": ["<name>"], "allow_fallbacks": false}.
• Build the pin map from the live endpoints API (precision ranking: fp32 > bf16/fp16 > fp8 > int8 > fp4/int4).
• Record the served provider on every response and verify the pin held.
• Unpinned results are not comparable between models or across time. If you inherit unpinned data, re-run - don't rescue it.

4. Deterministic checks beat the LLM judge's own flags

The judges' "fabricated" flags disagreed sharply with a deterministic check (is this %-or-multiplier actually present in the input?) - one model went from 7/9 judge-flagged to 0/10 deterministic, another direction entirely for a different model. Manual inspection sided with the deterministic check every time: the judges blur invented factual claims with acceptable illustrative numbers, and drift run to run.

• Wherever a property can be checked by a program - numbers-in-input, banned phrases, length, JSON validity, required fields - the program is the metric.
• Keep the judge's flag as advisory color only; never let it carry a claim in the report.
• Corollary: before trusting ANY judge-derived number, spot-check a sample by eye against the raw outputs.

5. Hard and trap cases are mandatory

An easy-only golden set produced "scoring is 100% solved across all models." It survived - but only after hard cases were added could that claim honestly be made, and the hard cases are what would have caught it if false.

Include, per structured task:

• Boundary cases - inputs sitting exactly on a decision edge (score ~= the tier cutoff). Expect dispersion; grade calibration, not a single right answer.
• False-trigger traps - inputs that superficially match a hard rule but don't actually (personal email BUT company identified; enterprise size BUT exec contact). These measure precision - a model that over-fires a disqualifier silently rejects your best leads.
• Buried positives - the rule genuinely applies but is hidden in distracting commercial-looking context.

And state what the traps do NOT test: "the traps test false positives; a subtly hidden true-DQ was not tested" - name the untested direction.

6. Blind the panel

• Anonymize candidates to letters; re-shuffle per case so position and label carry nothing.
• Judges from different model families than every candidate - same-family judges have correlated style priors.
• 3+ judges; aggregate across judges as part of n, and check no single judge drives the ranking.
• Fixed judging temperature (0) and a written rubric; the panel judges quality against the rubric, not "which do you like."

7. Small-sample honesty

The verdict from 8 cases x 5 seeds is a verdict about THESE prompts, THESE providers, THIS month.

• State N cases, N seeds, N judge-scores per model in the report, every time.
• Say "on these prompts" - not "model A is better than model B."
• Prices, rankings, and even served quantizations change monthly - date-stamp the report and treat it as perishable.
• Sanity-check alarming rates against their base: 0/10 fabrication on 10 samples is indistinguishable from a true 5% rate. Widen the sample before making a purity claim load-bearing.

The meta-rule

Before publishing any eval verdict, run the red-team pass the reference experiment ran: "what honest checks would embarrass this result?" Seeds, pins, hard cases, deterministic cross-checks - if any of them would plausibly flip the headline, do them first. The Round 2 -> Round 3 reversal happened because someone asked that question BEFORE publishing. Build the skill's reports so nobody has to ask.