Build custom TypeScript providers for prompt evaluation

This example demonstrates how to implement a custom API provider in TypeScript for promptfoo. The example includes a custom provider implementation in customProvider.ts and uses CSV test cases for prompt evaluation.

Use this example when you need to create a custom API provider for promptfoo. It provides a working template showing how to structure your custom provider code in TypeScript and how to use CSV files for test cases.

Do NOT use this if you're working with already-supported providers like OpenAI, Anthropic, or other mainstream APIs - the built-in providers will be simpler and better maintained. Avoid this approach if you don't need custom logic; standard provider configurations are sufficient for most use cases.

You provide a TypeScript file (customProvider.ts) that implements your custom provider logic, a prompts file (prompts.txt) containing the prompts to test, and a CSV file (vars.csv) with test case variables. The workflow outputs evaluation results, with results saved to output.json.

The example shows a command-line configuration that references both customProvider.js and openai:chat as providers.

This example is for TypeScript developers who need to implement custom providers for promptfoo and want to see a working example of how to structure the code and configuration.

To get started, initialize the example:

npx promptfoo@latest init --example provider-custom/typescript
cd provider-custom/typescript

Then run the evaluation:

promptfoo eval

For full control, you can specify all parameters explicitly:

promptfoo eval --prompts prompts.txt --tests vars.csv --providers openai:chat --output output.json --providers customProvider.js