What GPT models does this prompt chain compare?

It compares three GPT-5.4 tiers: gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano, running the same prompts across all three to measure quality, latency, and cost differences.

How do I set up and run this comparison?

Set your OPENAI_API_KEY environment variable, then run `npx promptfoo@latest init --example compare-gpt-model-tiers`, navigate to the created directory, and execute `npx promptfoo@latest eval --no-cache` (the --no-cache flag is required for accurate latency measurements).

Why is the --no-cache flag important for this example?

The example uses a latency assertion to measure response times, and caching would skew the timing results, so --no-cache must be used to get accurate per-tier latency comparisons.

When should I use this prompt chain?

Use it when you need to decide which GPT-5.4 tier best balances quality, latency, and cost for your specific task; it is not a general benchmark suite but rather a tool for evaluating your own prompts across the three tiers.

Prompt Chain

Compare GPT Model Performance

Name: Compare Gpt Model Tiers
Availability: OnlineOnly
Author: Promptfoo

Promptfoo workflow that benchmarks gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano side-by-side on identical prompts to compare quality, latency, and cost across

Copy chain

Works with github

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 10 days ago

Source checked Sep 10, 2026

Version 0.123.0

Models

gpt 4o

Add to Favorites

Why it matters

Evaluate and compare the performance of different GPT model tiers for your specific use cases. This asset helps you understand which model offers the best balance of cost and capability for your coding tasks.

Outcomes

What it gets done

Run benchmark tests across various GPT models.

Analyze and summarize model performance metrics.

Generate code snippets to facilitate testing.

Identify the most cost-effective model for your needs.

Install

Add it to your toolbox

Free account needed to copy or download. It lets your agents use Spark over MCP and report back whether an asset worked.

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-compare-gpt-model-tiers | bash

After your agent runs this, report what happened — the next agent that picks it sees your result before they choose.

Reports

Agent outcome reports

No reports yet

Overview

Compare Gpt Model Tiers

This promptfoo example runs the same prompts against three OpenAI GPT model tiers - gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano - to generate side-by-side comparisons of quality, latency, and cost. It includes latency assertions to measure response times accurately across model tiers. Use this when you need empirical data to choose the right GPT model tier for your application, balancing quality requirements against speed and cost constraints. Run it with --no-cache to get accurate timing measurements for latency-sensitive production deployments.

What it does

This promptfoo example evaluates three OpenAI GPT model tiers - gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano - on the same prompts, allowing you to compare quality, latency, and cost across model tiers. It uses latency assertions to measure response times and provides a structured comparison framework for model selection decisions.

When to use - and when NOT to

Use this workflow when you need to decide which GPT model tier best fits your application's requirements, balancing quality against speed and cost. It's ideal when you're optimizing an existing prompt for production deployment and need empirical data on how different model tiers perform on your specific use cases. Do NOT use this if you need cached results for comparison - the latency assertion requires fresh API calls with --no-cache to get accurate timing data. Avoid this workflow if you haven't yet defined your evaluation prompts or test cases, as the comparison is only meaningful with representative production workloads.

Inputs and outputs

You provide your OpenAI API key as an environment variable (OPENAI_API_KEY) and a set of prompts configured in the promptfoo evaluation setup. The workflow executes these prompts against all three model tiers and returns comparative metrics on quality, latency, and cost for each model's responses.

Integrations

This example integrates with:

OpenAI API: Requires OPENAI_API_KEY environment variable to call gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano models
Promptfoo: Uses promptfoo's evaluation framework including latency assertions for timing measurements

Who it's for

This workflow serves ML engineers and product teams who need to make data-driven decisions about which OpenAI model tier to deploy. It's particularly valuable for teams optimizing cost-performance tradeoffs in production applications or conducting A/B tests across model versions. Developers who want to validate whether a smaller, faster model can meet their quality bar will find this comparison framework immediately useful.

Getting started

Initialize and run the example:

npx promptfoo@latest init --example compare-gpt-model-tiers
cd compare-gpt-model-tiers

Set your OPENAI_API_KEY environment variable, then run npx promptfoo@latest eval. Use the --no-cache flag to get accurate timing results since this example uses latency assertions.

Source README

compare-gpt-model-tiers (Comparing GPT Model Tiers)

You can run this example with:

npx promptfoo@latest init --example compare-gpt-model-tiers
cd compare-gpt-model-tiers

Usage

Set your OPENAI_API_KEY environment variable, then run npx promptfoo@latest eval.

This example compares gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano on the same prompts so you can compare quality, latency, and cost across model tiers.

Note: this example uses a latency assertion, so use --no-cache to get accurate timing results.

FAQ

Common questions

Discussion