Compare OpenAI Model Performance

This Promptfoo example compares four OpenAI models - gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol, and gpt-6-astra - on riddles and puzzles, all run at the same low reasoning effort setting. gpt-6-astra requires model access on your own OpenAI account to run. It demonstrates side-by-side model comparison, cost and latency assertions to check performance thresholds, content validation via contains assertions to verify specific expected answers, and LLM-based grading via llm-rubric assertions for more nuanced evaluation criteria, across a diverse set of riddles testing different reasoning capabilities.

Use it as a template for comparing multiple OpenAI models on the same reasoning-heavy test set, or as a working example of combining deterministic assertions (content match, latency, cost) with LLM-graded rubric assertions in one Promptfoo config. It's a useful starting point if you want to evaluate which model handles a specific reasoning task best before committing to one in production.

Don't run the eval with response caching enabled - the example includes a latency assertion, and latency assertions don't support caching, so the eval must be run with --no-cache or the latency checks won't behave correctly.

Running npx promptfoo@latest eval --no-cache produces responses from all four models for each riddle, viewable side by side with npx promptfoo@latest view. An OPENAI_API_KEY must be set, either as an environment variable or in a .env file in the project directory.

Initialize the example directly from Promptfoo's example library:

npx promptfoo@latest init --example compare-openai-models

Developers evaluating which OpenAI model best handles a specific reasoning or puzzle-solving task, and anyone looking for a working example that mixes deterministic and LLM-rubric assertions in a single Promptfoo comparison.