Configure Tests Per Prompt
Run specific promptfoo test cases against only specific prompts instead of every prompt by default.
Why it matters
Streamline your prompt testing process by selectively running tests against specific prompts. This ensures that your test suite is efficient and relevant to the prompts being evaluated.
Outcomes
What it gets done
Define test configurations that target individual prompts.
Execute only the necessary tests for each prompt iteration.
Improve the efficiency of your prompt testing cycles.
Ensure test relevance by linking tests directly to prompts.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-config-tests-per-prompt | bash Steps
Steps in the chain
Overview
Config Tests Per Prompt
Shows how promptfoo's test-level prompts field overrides the default cartesian-product behavior, letting specific test cases run against only specific prompts by exact match or wildcard. Use when different prompts in a config need different test cases, rather than every test running against every prompt by default.
What it does
This example demonstrates running specific promptfoo test cases only against specific prompts using the test-level prompts field. By default promptfoo runs every test against every prompt (a cartesian product); the prompts field on a test case breaks that pattern and associates specific tests with specific prompts. Matching supports exact references by label or ID, wildcard patterns (e.g. Math:* matching Math:Basic and Math:Advanced), strict validation that errors at config load time on an invalid reference, and inheritance of a default prompt filter via defaultTest.
When to use - and when NOT to
Use this when different prompts in your config need different assertion criteria, you're testing prompt-specific behaviors, or you're organizing a large test suite that mixes several distinct prompt styles and don't want every test run against every prompt.
Not needed when every test is meant to run against every prompt - that's promptfoo's default cartesian-product behavior and needs no prompts field.
Inputs and outputs
Input is promptfooconfig.yaml defining labeled prompts (e.g. Factual Assistant, Creative Writer) and test cases with a prompts filter list. The example uses the built-in echo provider for demonstration; production configs would swap in a real LLM provider. Output is npx promptfoo@latest eval.
Integrations
Demonstrated with promptfoo's built-in echo provider; the prompts-field filtering mechanism itself works with any provider.
Who it's for
Teams with large or mixed-style prompt sets who need fine-grained control over which test cases run against which prompts.
npx promptfoo@latest init --example config-tests-per-prompt
Source README
config-tests-per-prompt (Filter Tests by Prompt)
This example demonstrates how to run specific tests only with specific prompts using the test-level prompts field.
Overview
By default, promptfoo runs each test against all prompts (a cartesian product). The prompts field on test cases lets you break this pattern and associate specific tests with specific prompts.
Use Cases
- Different assertion criteria for different prompt styles
- Testing prompt-specific behaviors
- Organizing large test suites with mixed prompt types
Quick Start
npx promptfoo@latest init --example config-tests-per-prompt
npx promptfoo@latest eval
Key Features
- Exact matching: Reference prompts by label or ID
- Wildcard patterns:
Math:*matchesMath:Basic,Math:Advanced - Strict validation: Invalid references error at config load time
- defaultTest inheritance: Set a default prompt filter for all tests
Configuration
The example uses the echo provider for demonstration. In production, replace with your preferred LLM provider.
prompts:
- id: factual
label: Factual Assistant
raw: 'You are factual. Answer: {{question}}'
- id: creative
label: Creative Writer
raw: 'Be creative. Answer: {{question}}'
tests:
- vars:
question: 'What is 2+2?'
prompts:
- Factual Assistant # Only runs with this prompt
assert:
- type: contains
value: '4'
Learn More
See the Filtering Tests by Prompt documentation for more details.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.