Generate Configurable Python Test Cases from Templates
Dynamically generate Python test cases with configurable parameters, enabling reusable test functions customized through YAML configuration.
Why it matters
Enable developers to dynamically generate Python test cases with customizable parameters, allowing reusable test generation functions that can be configured differently for various testing scenarios without code changes.
Outcomes
What it gets done
Pass configuration objects to Python test generator functions
Control test case generation with parameters like language lists and row limits
Generate test cases from CSV files with configurable row constraints
Maintain backward compatibility with legacy test generator formats
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-config-python-test-cases | bash Steps
Steps in the chain
Overview
Config Python Test Cases
This example demonstrates how to use Python functions to generate test cases with configurable parameters using the TestGeneratorConfig feature. Use this when you need to generate multiple variations of test cases from a single Python function by passing different configuration parameters, such as specifying different language pairs or limiting the number of test cases generated from CSV files.
What it does
This example demonstrates how to use Python functions to generate test cases with configurable parameters using the TestGeneratorConfig feature. It allows you to pass configuration objects to customize test generation, enabling you to control languages, row limits, and other parameters while maintaining backward compatibility with existing test generators.
When to use - and when NOT to
Use this when you need to generate multiple variations of test cases from a single Python function by passing different configuration parameters, such as specifying different language pairs or limiting the number of test cases generated from CSV files. Use it when you want reusable test generation functions that can be customized without code changes. Do not use this if you only need static test cases that never change, as simple YAML-defined tests would be more straightforward. Do not use this if you don't have Python installed or cannot manage Python dependencies.
Inputs and outputs
You provide Python functions that accept an optional config parameter and a promptfoo configuration YAML file that references these functions with configuration objects specifying parameters like languages or max_rows. For example:
tests:
- path: file://test_cases.py:generate_simple_tests
config:
languages: [German, Italian]
The Python function receives the config dictionary:
from typing import Optional, Dict, Any
def generate_simple_tests(config: Optional[Dict[str, Any]] = None):
languages = ["Spanish", "French"] # defaults
if config:
languages = config.get("languages", languages)
# Generate test cases using the configuration...
Integrations
This example works with pandas for CSV-based test case generation. The test generator functions are called by promptfoo during evaluation runs.
Who it's for
This is for prompt engineers and QA teams who need to generate test cases programmatically with varying parameters. The backward compatibility feature means existing promptfoo users can adopt this incrementally without breaking current workflows.
Setup
Install the example:
npx promptfoo@latest init --example config-python-test-cases
cd config-python-test-cases
Install Python dependencies:
pip install pandas
Set your API key:
export OPENAI_API_KEY=your_api_key_here
Run the evaluation:
promptfoo eval
Source README
config-python-test-cases (Python Test Cases with Configuration)
You can run this example with:
npx promptfoo@latest init --example config-python-test-cases
cd config-python-test-cases
This example demonstrates how to use Python functions to generate test cases with configurable parameters using the new TestGeneratorConfig feature.
Overview
Previously, test generators could only be called without parameters:
tests:
- file://test_cases.py:generate_simple_tests
Now you can pass configuration objects to customize the test generation:
tests:
- path: file://test_cases.py:generate_simple_tests
config:
languages: [German, Italian]
Requirements
Python Dependencies:
pip install pandasEnvironment Variables:
export OPENAI_API_KEY=your_api_key_here
Usage
Run the evaluation with:
promptfoo eval
Features Demonstrated
1. Backward Compatibility
The old format still works:
- file://test_cases.py:generate_simple_tests
2. Simple Configuration
Pass configuration to customize test generation:
- path: file://test_cases.py:generate_simple_tests
config:
languages: [German, Italian]
3. Row Limiting
Control how many test cases are generated:
- path: file://test_cases.py:generate_from_csv
config:
max_rows: 2
Implementation
The Python functions accept an optional config parameter:
from typing import Optional, Dict, Any
def generate_simple_tests(config: Optional[Dict[str, Any]] = None):
languages = ["Spanish", "French"] # defaults
if config:
languages = config.get("languages", languages)
# Generate test cases using the configuration...
This enables:
- Backward compatibility: Existing generators work unchanged
- Flexible configuration: Pass any parameters as JSON
- Reusable functions: Same function, different configurations
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.