When should I use config prompts per model instead of the default promptfoo behavior?

Use this when different models in your eval need differently-worded or differently-structured prompts—for example, when a model responds better to a specific instruction style. The default promptfoo behavior runs the same prompt across every configured model, so this approach is only needed when you want model-specific prompt variations.

What do I need to set up to use this prompt chain?

You need to edit `promptfooconfig.yaml` to map specific prompts to specific models, and provide your `OPENAI_API_KEY`. You can initialize the example with `npx promptfoo@latest init --example config-prompts-per-model`.

Does this prompt chain work with providers other than OpenAI?

Yes, the per-model prompt mapping works with any provider set. The example uses OpenAI via `OPENAI_API_KEY`, but the same approach applies to other providers.

Prompt Chain

Configure Prompts for Multiple Models

Name: Config Prompts Per Model
Availability: OnlineOnly
Author: Promptfoo

Set up a custom set of promptfoo prompts targeted at specific models within a single eval.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Add to Favorites

Why it matters

Streamline your AI development by configuring prompts specifically for different language models. This asset helps manage and apply model-specific prompt variations efficiently.

Outcomes

What it gets done

Define prompts tailored to individual LLM capabilities.

Organize and select appropriate prompts based on the target model.

Ensure consistent and effective prompt usage across various AI models.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-config-prompts-per-model | bash

Steps

Steps in the chain

Initialize example project

Navigate to project directory

Set OpenAI API key

Edit configuration file

Run evaluation

View results

Overview

Config Prompts Per Model

Shows how to assign specific prompts to specific models within a single promptfoo eval, instead of running one shared prompt across all models. Use when different models being compared need differently-worded prompts rather than one prompt shared across all of them.

What it does

This example shows how to set up a custom set of prompts for specific models within a single promptfoo eval, rather than running the same prompt across every configured model.

When to use - and when NOT to

Use this when different models in your eval need differently-worded or differently-structured prompts - for example, a model that responds better to a specific instruction style.

Not needed when the same prompt is meant to be compared identically across all models - that's the promptfoo default cartesian-product behavior.

Inputs and outputs

Input is promptfooconfig.yaml (edited to map specific prompts to specific models) plus OPENAI_API_KEY. Output is a normal promptfoo eval / promptfoo view run.

Integrations

Runs against OpenAI via OPENAI_API_KEY in this example; the per-model prompt mapping works with any provider set.

Who it's for

Teams comparing multiple models where each model needs its own tailored prompt rather than one shared prompt.

npx promptfoo@latest init --example config-prompts-per-model
cd config-prompts-per-model

Source README

config-prompts-per-model (Prompts Per Model)

You can run this example with:

npx promptfoo@latest init --example config-prompts-per-model
cd config-prompts-per-model

Usage

This example shows how to set up a custom set of prompts for specific models.

To get started, set your OPENAI_API_KEY environment variable.

Next, edit promptfooconfig.yaml.

Then run:

promptfoo eval

Afterwards, you can view the results by running promptfoo view

FAQ

Common questions

Discussion