Compare Code Skills and Generate Tests
Promptfoo example A/B testing two versions of a Codex review skill with max-score for objective winner selection.
Why it matters
Automate the evaluation of AI coding capabilities by comparing their performance on specific tasks and generating relevant test cases.
Outcomes
What it gets done
Compare the output of different coding models against a reference.
Generate unit tests to verify code correctness.
Identify discrepancies and potential issues in generated code.
Automate code review and testing processes.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-skill-comparison | bash Steps
Steps in the chain
Overview
Skill Comparison
A Promptfoo example A/B testing two versions of a Codex review-standards skill on identical tasks, using a shared output schema and max-score to objectively select the stronger version by issue recall and precision. Use when A/B testing two versions of the same Codex skill against identical tasks with objective scoring.
What it does
A Promptfoo example that compares two versions of the same Codex review-standards skill against identical code-review tasks, using the max-score assertion to pick a winner. The v1 fixture only flags weak password hashing; the v2 fixture also checks for timing-safe secret comparison, a strictly broader review scope. Both provider versions share a single output_schema, declared once via a YAML anchor, so every response is guaranteed to match the same review JSON shape, making the two versions directly comparable. The eval verifies which skill was actually used (skill-used), scores each candidate's issue recall and precision, and uses max-score to select the better output per test case - since v2 is a strict superset of v1's checks, the weaker v1 candidate is expected to lose whenever a test case exercises the timing-safe-comparison check that only v2 catches.
When to use - and when NOT to
Use as a reference for A/B testing two versions of the same Codex skill against the same tasks, with an objective scoring method rather than manual comparison. Not the right example for comparing entirely different skills or models - it's specifically structured around two versions of one skill sharing one output schema so their outputs are directly scorable against each other.
Inputs and outputs
Input is the two skill fixture directories, fixtures/v1 and fixtures/v2, or their absolute paths via environment variables if run from elsewhere, plus either an API key or a local Codex login referenced via CODEX_HOME_OVERRIDE. Output, after running the eval with caching disabled, is a per-test-case comparison showing which skill was used, its issue recall and precision scores, and which version's output was selected as the max-score winner.
npx promptfoo@latest init --example openai-codex-sdk/skill-comparison
cd openai-codex-sdk/skill-comparison
Integrations
Built on the OpenAI Codex SDK via Promptfoo, using Promptfoo's max-score assertion for objective winner selection and a shared YAML-anchored output_schema to keep both skill versions' outputs directly comparable.
Who it's for
Developers iterating on a Codex skill's review coverage who want to prove a new version is objectively stronger than the last, using scored recall and precision rather than eyeballing outputs.
Source README
openai-codex-sdk/skill-comparison (Codex Skill Comparison)
You can run this example with:
npx promptfoo@latest init --example openai-codex-sdk/skill-comparison
cd openai-codex-sdk/skill-comparison
Overview
This example compares two versions of the same Codex skill against identical review tasks.
fixtures/v1contains a narrowerreview-standardsskill that only calls out weak password hashing.fixtures/v2contains a stronger version that also checks timing-safe secret comparison.- Both providers share an
output_schema(declared once via a YAML anchor) so each response is guaranteed to match the review JSON shape. - The eval verifies
skill-used, scores issue recall and precision, and usesmax-scoreto select the best output for each test case.
Run it from this directory with:
promptfoo eval --no-cache
If you run it from another directory, set these environment variables first:
export CODEX_SKILL_COMPARE_V1_DIR=/absolute/path/to/fixtures/v1
export CODEX_SKILL_COMPARE_V2_DIR=/absolute/path/to/fixtures/v2
The checked-in sample-codex-home directory is intentionally empty of auth state. Use an API key, or set CODEX_HOME_OVERRIDE="$HOME/.codex" to reuse a local Codex login.
Because this example uses max-score, the weaker candidate is expected to fail when Promptfoo marks the stronger output as the winner for a test case.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.