4,022 tools found
Promptfoo example using shell scripts, Python or any binary as dynamic prompt generators via the exec: prefix or auto-detection.
Promptfoo example using the extensions API - beforeAll/afterAll/beforeEach/afterEach hooks in Python or JS for custom test setup/teardown.
Promptfoo example loading defaultTest config from an external file via file:// syntax, for sharing test defaults across projects.
Promptfoo example generating test cases from TypeScript/JavaScript functions - static typed cases plus dynamic ones simulated from a database.
Promptfoo example evaluating multi-turn conversations using the built-in _conversation variable to reference prior prompts and outputs.
Promptfoo workflow example demonstrating how to organize and run multiple prompt sets against different providers in a single evaluation using prompt labels.
Promptfoo workflow example that demonstrates how to configure different prompt sets for specific AI models.
Python test case generator for promptfoo that accepts configuration parameters to customize test generation with language options and row limits.
Promptfoo example using extension hooks to send metrics, trigger alerts, and integrate eval results with CI/CD.
A promptfoo test harness with intentional 50% error rate to validate retry functionality using the --retry-errors flag in prompt evaluation workflows.
Promptfoo workflow example demonstrating how to use paths and wildcards to simplify promptfooconfig.yaml structure.
Promptfoo workflow example named 'Store And Reuse Outputs' that can be initialized and run using promptfoo commands with OpenAI API key configuration.
