Generate Configurable Python Test Cases from Templates

Config Python Test Cases is a promptfoo example that demonstrates how to use Python functions to generate test cases with configurable parameters using the TestGeneratorConfig feature. It allows you to pass configuration objects to customize test generation, such as specifying languages or limiting the number of rows, while maintaining backward compatibility with existing test generators.

Use this when you need to generate multiple variations of test cases from a single Python function by passing different configuration parameters, such as testing translation prompts across different languages or controlling the volume of test cases generated from CSV files. Use it when you want to reuse the same test generation logic with different settings without duplicating code.

Do not use this if you need only static test cases without dynamic generation, or if your test generation logic does not require any parameterization.

You provide Python functions that accept an optional config parameter containing customization options, and YAML configuration files that specify which test generator functions to call and what parameters to pass. Examples from the source show passing language arrays like [German, Italian] and row limits like max_rows: 2.

You receive dynamically generated test cases based on your configuration parameters, which promptfoo then uses to evaluate your prompts.

This example integrates with:

• promptfoo: The core evaluation framework that runs the test cases
• OpenAI API: Requires an API key for prompt evaluation
• pandas: Python library required for CSV-based test generation
• Python: Test generator functions written in Python with typing support

Initialize the example:

npx promptfoo@latest init --example config-python-test-cases
cd config-python-test-cases

Install Python dependencies:

pip install pandas

Set your API key:

export OPENAI_API_KEY=your_api_key_here

Configure test generation with parameters:

tests:
  - path: file://test_cases.py:generate_simple_tests
    config:
      languages: [German, Italian]

Or limit rows from CSV:

- path: file://test_cases.py:generate_from_csv
  config:
    max_rows: 2

Run the evaluation:

promptfoo eval

The Python implementation accepts optional configuration:

from typing import Optional, Dict, Any

def generate_simple_tests(config: Optional[Dict[str, Any]] = None):
    languages = ["Spanish", "French"]  # defaults

    if config:
        languages = config.get("languages", languages)

    # Generate test cases using the configuration...

This is for prompt engineers and QA teams who need to generate parameterized test cases for LLM evaluation workflows. It is particularly useful for teams testing multilingual prompts or those who want to control test case volume dynamically. The backward compatibility feature makes it suitable for teams migrating from simpler test generation patterns to more configurable approaches.