Organize and label prompts across multiple providers

This prompt chain demonstrates how to use prompt labels in Promptfoo to organize and execute evaluation workflows. It enables you to run one set of multiple prompts against a specific provider, another set of prompts against a different provider, and consolidate all results into a single evaluation output.

Use this workflow when you need to test different prompt variations against multiple LLM providers simultaneously or organize evaluations with distinct prompt groups. This approach allows you to manage multiple prompt-provider combinations within a single configuration.

Do NOT use this if you only need to test a single prompt against one provider - the labeling overhead adds unnecessary complexity for simple evaluations.

You provide a promptfooconfig.yaml file that defines your prompt sets with labels, specifies which providers to use, and maps which labeled prompts run against which providers. You also need to set your OPENAI_API_KEY environment variable.

You receive a single evaluation result from all prompt-provider combinations, viewable through the Promptfoo interface by running promptfoo view.

This workflow requires the Promptfoo evaluation framework and an OpenAI API key. It uses Promptfoo's configuration system via YAML files and the command-line interface for execution and result visualization.

To get started:

npx promptfoo@latest init --example config-prompt-labels
cd config-prompt-labels

After editing your configuration, run the evaluation:

promptfoo eval

View results with promptfoo view to launch the interface.

This workflow serves ML engineers and prompt engineers who need to organize and execute evaluations across multiple prompts and providers. The labeling system allows you to manage different prompt sets and provider combinations within a single configuration file.