Compare GPT-4o and GPT-4o Mini Performance

This Promptfoo config compares openai:gpt-5 against openai:gpt-5-mini on riddle-solving, testing 15 riddles via a shared prompt template ("Solve this riddle: {{riddle}}"). Three riddles carry explicit correctness checks: an echo riddle ("I speak without a mouth and hear without ears...") checked for the word "echo" plus an llm-rubric instruction ("do not apologize"); a boat riddle checked via llm-rubric for explaining that everyone aboard is married, so no "single" people remain; and a riddle about darkness ("the more of this there is, the less you see") checked for the word "darkness".

The remaining 12 riddles carry no per-test assertion, relying only on a defaultTest rule applied to every case: each response must cost under $0.002 and return in under 3000ms. These span a wide range of classic wordplay: a keyboard riddle ("keys but no locks... space but no room"), a fire riddle ("not alive but grows... water kills me"), a postage-stamp riddle ("travel around the world while staying in a corner"), a wordplay riddle built on the word "ton" reading as "not" backward, a coffin riddle ("the person who makes it sells it, the person who buys it never uses it"), a joke riddle ("can be cracked, made, told, and played"), a piano riddle ("has keys but can't open locks"), a cloud riddle ("fly without wings, cry without eyes"), a pencil-lead riddle ("taken from a mine, shut in a wooden case, never released"), a trick question about the third of "David's father's" three sons - the answer is David himself - and a breath riddle ("light as a feather, yet the strongest person can't hold me for much more than a minute") that appears twice with slightly different phrasing, a near-duplicate pair within the test set.

Use it as a template for comparing a full-size and mini model tier's reasoning quality on wordplay and riddle tasks, under a shared cost and latency budget. Do not use it if you need broader reasoning benchmarks beyond riddles, or if cost and latency ceilings this tight don't match your production budget.

Input: the YAML config - the riddle prompt template, the two provider IDs, the global cost/latency defaultTest, and per-test assertions where present. Output: Promptfoo's evaluation report comparing the two models' pass rates on correctness, where asserted, plus cost and latency across all 15 riddles.

Uses Promptfoo's native OpenAI provider for both models, plus its contains, llm-rubric, cost, and latency assertion types.

Teams choosing between a full-size and mini OpenAI model tier who want to compare reasoning quality on a fixed cost and latency budget.