3,719 tools found
Organize promptfoo test configurations using paths and wildcards to simplify large test suites and improve maintainability.
Persist and reuse prompt evaluation outputs across runs to build efficient multi-phase testing pipelines and reduce API costs.
A promptfoo example enforcing JSON-schema structured outputs across OpenAI, Azure OpenAI, and Anthropic.
Run specific promptfoo test cases against only specific prompts instead of every prompt by default.
Runs isolated test suites per provider in promptfoo via the providers filter, avoiding the default test x provider cross-product.
A promptfoo example demonstrating how to build and test custom API providers with CSV test cases for prompt evaluation workflows.
Promptfoo HTTP provider example for OpenAI streaming - and why streaming targets do not help evaluations.
Three patterns for overriding promptfoo's default assertion scoring function in config, JavaScript, or Python.
Measures semantic similarity between LLM outputs and reference text using BERTScore in a promptfoo assertion.
Evaluate whether chatbot responses stay relevant across a conversation using promptfoo's sliding-window conversation-relevance assertion.
Example chain that shows how to differentiate between provider errors and failed assertions in LLM testing workflows.
Evaluate GPT-4o-mini's zero-shot IMDB sentiment classification with precision, recall, and F1 metrics in promptfoo.
