This is a promptfoo evaluation configuration that benchmarks two language models against a battery of riddles, despite its title naming GPT-4o and GPT-4o Mini - the actual providers list in the config specifies openai:gpt-5 and openai:gpt-5-mini, a title/config mismatch preserved here as-is rather than silently corrected. It runs a single prompt template ("Solve this riddle: {{riddle}}") across fourteen riddle variations, applying a defaultTest assertion block to every case: cost must stay under $0.002 per inference and latency under 3000ms, functioning as guardrails on operational cost and speed alongside accuracy.

Several riddles carry specific per-test assertions beyond the defaults: a "speak without a mouth" riddle requires the output to contain the word "echo" and, via an llm-rubric model-graded assertion, requires the model not to apologize; a boat riddle requires the rubric to confirm the answer explains that everyone aboard is married (so no "single" people); a darkness riddle requires the literal word "darkness" in the output. The remaining ten riddles - keys-without-locks, growing-without-lungs, "travels around the world in a corner," "heavy forward, light backward," and others including a "Snap, Crackle, and ___" pop-culture riddle - have no per-test assertions beyond the cost/latency defaults, relying on manual or comparative review of the two models' answers rather than automated correctness grading.

Use this configuration when comparing two language models' performance on riddle-solving and reasoning tasks, or as a template for building similar promptfoo comparisons with cost/latency thresholds and a mix of exact-match, contains, and LLM-rubric assertions.

It is not a general-purpose benchmark suite - it's scoped narrowly to riddle-solving with a specific pair of models configured (currently GPT-5 and GPT-5 Mini despite the title), and its default cost/latency thresholds are tuned for that specific model pairing rather than universal.

Input is a set of riddle strings substituted into the shared prompt template, run against both configured providers. Output is promptfoo's standard evaluation report comparing each model's answers per riddle, pass/fail status against the cost/latency thresholds and any per-test content/rubric assertions, and a side-by-side view for the riddles without automated grading.

providers:
  - openai:gpt-5
  - openai:gpt-5-mini

Built on the promptfoo evaluation framework's YAML config schema, using its cost, latency, contains, and llm-rubric assertion types, and targeting OpenAI's API for both compared models.

Developers or teams evaluating LLM reasoning and riddle-solving performance plus cost/latency tradeoffs between two models, using promptfoo as the evaluation harness.