1,210 tools found
Dynamically generate Python test cases with configurable parameters, enabling reusable test functions customized through YAML configuration.
Hook into AI evaluation lifecycle to process results, send metrics, trigger alerts, and integrate with CI/CD pipelines programmatically.
Organize promptfoo test configurations using paths and wildcards to simplify large test suites and improve maintainability.
A promptfoo example enforcing JSON-schema structured outputs across OpenAI, Azure OpenAI, and Anthropic.
A Promptfoo example showing how the test-level prompts field limits specific tests to specific prompts, instead of the default cartesian product.
Custom provider example for promptfoo that demonstrates how to integrate a custom API provider with CSV test cases for prompt evaluation workflows.
A promptfoo example calling OpenAI's streaming API via HTTP provider - and why promptfoo recommends against streaming in evals.
Measures semantic similarity between LLM outputs and reference text using BERTScore in a promptfoo assertion.
Promptfoo example evaluating GPT-4o vision models on Fashion MNIST image classification with a structured schema.
Run external JavaScript assertions to validate AI-generated code quality and correctness with automated testing workflows.
Promptfoo example workflow for evaluating JSON output from language models.
Promptfoo max-score assertion for objective, weighted output selection - deterministic, no extra LLM calls.
