Compare GPT Model Performance

This prompt chain compares three OpenAI GPT model tiers (gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano) by running identical prompts across all three and measuring quality, latency, and cost. It uses promptfoo's evaluation framework with latency assertions to provide accurate timing data for model performance comparisons.

Use this workflow when you need to decide which GPT model tier fits your application's requirements, especially when balancing response quality against speed and API costs. It's ideal when prototyping a new feature and you're unsure whether the full model's capabilities justify the expense, or when optimizing an existing system to reduce latency or costs without sacrificing output quality.

Do NOT use this if you need cached results for comparison - the latency assertion requires running with --no-cache to get accurate timing measurements, so repeated evaluations will consume API credits. Avoid this approach if you're comparing models outside the gpt-5.4 family, as the example is specifically configured for these three tiers.

You provide your OpenAI API key as an environment variable (OPENAI_API_KEY) and the same set of prompts that will be evaluated across all three model tiers. The workflow outputs comparative metrics for quality, latency, and cost across gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano, allowing you to make data-driven decisions about which model tier to deploy.

This workflow integrates with:

• OpenAI API: Connects to gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano models for evaluation
• promptfoo: Uses the promptfoo evaluation framework for running comparisons and latency assertions

To get started, run:

npx promptfoo@latest init --example compare-gpt-model-tiers
cd compare-gpt-model-tiers

Then set your OPENAI_API_KEY environment variable and execute npx promptfoo@latest eval to run the comparison.

This workflow is built for ML engineers and product teams evaluating which OpenAI model tier to use in production. It's particularly valuable for developers building cost-sensitive applications who need quantitative data to justify model selection, and for engineering managers who need to demonstrate the trade-offs between model performance and infrastructure costs to stakeholders.