Compare LLM Reasoning Effort

This prompt chain compares the same gpt-5.4-mini Responses API model with two different reasoning effort settings: none for low-latency tasks that do not need reasoning tokens, and medium for tasks where extra deliberation may improve reliability. The evaluation keeps the provider, prompt, verbosity, and output limit identical across both configurations, making it easy to isolate and compare output quality, latency, and cost for each reasoning effort level.

Use this example when you need to decide whether enabling reasoning tokens justifies the added latency and cost for your specific use case. It is ideal when you are evaluating whether your tasks benefit from extra deliberation (medium reasoning) or can rely on faster, direct responses (no reasoning). Do NOT use this if you are working with models other than gpt-5.4-mini or if you need to compare across different model families - this example is specifically designed to isolate the reasoning effort variable within a single model.

You provide an OpenAI API key via the OPENAI_API_KEY environment variable. The example runs a set of evaluation prompts against both reasoning configurations. You receive a side-by-side evaluation table that displays output quality, latency metrics, and cost data for each reasoning effort setting, allowing direct comparison of the trade-offs.

This example integrates with:

• Promptfoo: The evaluation framework that orchestrates the comparison, runs the tests, and generates the evaluation table
• OpenAI API: Specifically the gpt-5.4-mini model via the Responses API, which supports configurable reasoning effort levels

This example is for AI engineers and product teams who are optimizing their OpenAI API usage and need empirical data to choose the right reasoning effort setting. It is particularly valuable for developers who want to understand the performance and cost implications of reasoning tokens before deploying gpt-5.4-mini in production. If you are building latency-sensitive applications or working within tight budget constraints, this comparison helps you make an informed decision about whether the quality improvements from medium reasoning justify the additional overhead.

To run this example, initialize it with:

npx promptfoo@latest init --example compare-gpt-reasoning-effort
cd compare-gpt-reasoning-effort

Set your OPENAI_API_KEY environment variable, then execute the evaluation:

promptfoo eval --no-cache

The --no-cache flag ensures fresh API calls for accurate latency and cost measurements.