Integrate custom API providers with promptfoo testing
A promptfoo example demonstrating how to build and test custom API providers with CSV test cases for prompt evaluation workflows.
Why it matters
Enable developers to test and evaluate prompts using custom API providers alongside standard providers like OpenAI, allowing flexible integration of proprietary or specialized language model endpoints into the promptfoo evaluation framework.
Outcomes
What it gets done
Configure custom JavaScript provider modules for prompt evaluation
Run evaluations combining custom providers with standard OpenAI endpoints
Load test cases from CSV files for systematic prompt testing
Output evaluation results to JSON for analysis and comparison
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-basic | bash Overview
Basic
This example demonstrates how to create and test a custom API provider within promptfoo's evaluation framework using CSV test cases. Use this when you need to evaluate prompts against a custom provider implementation. Skip it if you're using natively supported providers.
What it does
This example demonstrates how to create and test a custom API provider within promptfoo's evaluation framework. It shows how to integrate custom logic into promptfoo's testing pipeline, using CSV files to define test cases and variables for systematic prompt evaluation.
When to use - and when NOT to
Use this example when you need to evaluate prompts against a custom provider that isn't natively supported by promptfoo. Do NOT use this if you're working with standard OpenAI, Anthropic, or other natively supported providers - use promptfoo's built-in provider configurations instead for simpler setup.
Inputs and outputs
You provide a custom provider implementation in JavaScript (customProvider.js), a prompts file (prompts.txt), and test cases in CSV format (vars.csv). The framework executes your prompts against the custom provider using the variables from your CSV, then outputs evaluation results to output.json. The CSV structure allows you to define multiple test scenarios with different input variables that get systematically tested.
Integrations
The custom provider architecture allows integration with any API or service you can call from Node.js. The example command line shows the custom provider being used alongside other providers in the evaluation.
Who it's for
This example is for developers who need to evaluate prompts using custom providers. This approach requires JavaScript development skills to implement the custom provider interface.
Getting started
Initialize the example with:
npx promptfoo@latest init --example provider-custom/basic
cd provider-custom/basic
Run the evaluation:
promptfoo eval
Or use the full command-line syntax:
promptfoo eval --prompts prompts.txt --tests vars.csv --providers openai:chat --output output.json --providers customProvider.js
Source README
provider-custom/basic (Custom Provider)
You can run this example with:
npx promptfoo@latest init --example provider-custom/basic
cd provider-custom/basic
Usage
This example uses a custom API provider in customProvider.js. It also uses CSV test cases.
Run:
promptfoo eval
Full command-line equivalent:
promptfoo eval --prompts prompts.txt --tests vars.csv --providers openai:chat --output output.json --providers customProvider.js
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.