What metrics does this comparison evaluate?

It evaluates correctness (using contains assertions), nuance (using llm-rubric assertions), cost, and latency for each model's responses.

How do I set up and run this prompt chain?

Run `npx promptfoo@latest init --example compare-openai-models`, set your OPENAI_API_KEY in a .env file or directly, then execute `npx promptfoo@latest eval --no-cache`. View results with `npx promptfoo@latest view`.

Prompt Chain

Compare OpenAI Model Performance

Name: Compare Openai Models
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example comparing GPT-5.4 and GPT-5.4 Mini on riddles with cost, latency, and LLM-graded quality assertions.

Copy chain

Works with openai

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

gpt 4o

Add to Favorites

Why it matters

Evaluate and compare the performance of different OpenAI language models on reasoning and riddle-solving tasks. Understand which models excel at specific challenges.

Outcomes

What it gets done

Run comparative tests across multiple OpenAI models.

Analyze model responses to riddles and reasoning problems.

Generate insights into model strengths and weaknesses.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-compare-openai-models | bash

Steps

Steps in the chain

Initialize the example

Navigate to project directory

Set OpenAI API key

Run the evaluation

View the results

Overview

Compare Openai Models

This example compares OpenAI's GPT-5.4 and GPT-5.4 Mini on riddles and reasoning tasks, scoring correctness, cost, and latency. Use it when deciding between GPT-5.4 and GPT-5.4 Mini for a reasoning task; the latency assertion requires running with --no-cache.

What it does

This example compares OpenAI's gpt-5.4 with gpt-5.4-mini across various riddles and reasoning tasks, using content validation, cost and latency assertions, and LLM-based grading.

When to use - and when NOT to

Use it when you want to decide between GPT-5.4 and GPT-5.4 Mini for a reasoning-style task, weighing correctness against cost and latency, or as a working example of combining contains and llm-rubric assertions. Do not run it with caching enabled - the example uses a latency assertion that doesn't support caching, so --no-cache is required.

Inputs and outputs

Requires OPENAI_API_KEY. Running npx promptfoo@latest eval --no-cache produces both models' responses to the same riddles side by side, scored for correctness (contains), nuance (llm-rubric), cost, and latency; results are viewable via npx promptfoo@latest view, showing both models' answers to each riddle side by side for direct comparison.

Integrations

Set up via npx promptfoo@latest init --example compare-openai-models, then cd into the created directory, set OPENAI_API_KEY directly or in a .env file, then run with --no-cache.

Who it's for

Teams deciding between GPT-5.4 and GPT-5.4 Mini for reasoning tasks, using a diverse set of riddles that test different reasoning capabilities to compare correctness alongside real cost and latency trade-offs.

Source README

compare-openai-models (OpenAI Model Comparison)

This example compares OpenAI's gpt-5.4 with gpt-5.4-mini across various riddles and reasoning tasks.

You can run this example with:

npx promptfoo@latest init --example compare-openai-models
cd compare-openai-models

Quick Start

Initialize this example by running:

npx promptfoo@latest init --example compare-openai-models

Navigate to the newly created compare-openai-models directory:
```
cd compare-openai-models
```
Set an OpenAI API key directly in your environment:
```
export OPENAI_API_KEY="your_openai_api_key"
```
Alternatively, you can set the API key in a .env file:
```
OPENAI_API_KEY=your_openai_api_key
```
Run the evaluation with:
```
npx promptfoo@latest eval --no-cache
```
Note: the --no-cache flag is required because the example uses a latency assertion which does not support caching.
View the results:
```
npx promptfoo@latest view
```
The expected output will include the responses from both models for the provided riddles, allowing you to compare their performance side by side.

What this example demonstrates

This example compares OpenAI's GPT-5.4 with GPT-5.4 Mini across various riddles and puzzles. It demonstrates:

Model comparison: Side-by-side evaluation of gpt-5.4 vs gpt-5.4-mini
Cost and latency assertions: Ensuring responses meet performance thresholds
Content validation: Using contains assertions to verify specific answers
LLM-based grading: Using llm-rubric assertions for nuanced evaluation criteria
Diverse test cases: A variety of riddles testing different reasoning capabilities

FAQ

Common questions

Discussion