Compare Model Outputs Across Prompts

This Promptfoo config demonstrates the per-provider prompts field, which restricts which prompt templates run against which model rather than running every prompt against every provider by default. Two prompts are defined, loaded from external prompt1.txt and prompt2.txt files and labeled "My first prompt" and "My second prompt". gpt-4.1-mini is configured to run both prompts, while gpt-4 is restricted to only the first prompt - so the eval matrix is asymmetric rather than the full cross-product of prompts and providers.

Sixteen test cases each supply a different topic variable, ranging from playful ones like "bananas" and "midnight snacking" to substantive ones like "the ethical implications of AI" and "the impact of deepfakes on societal trust", each run against whichever prompt and provider combinations are configured.

Use it as a template when different models need different prompt variants - for example, a longer, more explicit prompt for a weaker model and a terser one for a stronger model - rather than testing every prompt against every provider. Do not use it if all your providers should be tested against the same full set of prompts, which is Promptfoo's default behavior already.

Input: the YAML config - two external prompt files and per-provider prompts lists restricting which prompts each model runs - plus 16 topic variables. Output: Promptfoo's evaluation report, with gpt-4.1-mini results for both prompts and gpt-4 results for only the first, side by side per topic.

Uses Promptfoo's per-provider prompts field alongside its external prompt-file loading (file://) and prompt labeling.

Teams that need to test different, per-model prompt variants against different providers rather than a uniform prompt set applied to every provider.