Build custom TypeScript providers for prompt evaluation

This example demonstrates how to implement a custom API provider in TypeScript for promptfoo. It includes a custom provider implementation in customProvider.ts and uses CSV-based test cases for prompt evaluation.

Use this example when you want to create a custom provider in TypeScript for promptfoo. It provides a starting point for implementing your own provider logic.

Do NOT use this if standard promptfoo providers already support your target API - the built-in providers offer better maintenance and feature support. Avoid this approach if you're just starting with promptfoo and haven't yet mastered the standard evaluation workflow.

You provide a custom TypeScript provider implementation in customProvider.ts, a prompts file (prompts.txt), and test cases in CSV format (vars.csv). The example accepts standard promptfoo configuration including prompt templates and variable substitutions from the CSV.

You receive evaluation results with output saved to output.json.

Initialize the example with:

npx promptfoo@latest init --example provider-custom/typescript
cd provider-custom/typescript

Run the evaluation:

promptfoo eval

For the full command-line equivalent with explicit parameters:

promptfoo eval --prompts prompts.txt --tests vars.csv --providers openai:chat --output output.json --providers customProvider.js

The example uses promptfoo's evaluation framework for test execution and result collection.

This example serves TypeScript developers who want to build custom providers for promptfoo. It's designed for developers who need to implement their own provider logic beyond what standard providers offer.