Evaluate prompts with custom JavaScript providers

Mjs is a working example that demonstrates how to implement a custom API provider for promptfoo using JavaScript modules (.mjs format). It provides a reference implementation with a custom provider file (customProvider.mjs), prompt definitions, and CSV-based test cases.

Use this example when you need to create a custom provider implementation for promptfoo. It demonstrates the structure and setup for custom providers using .mjs modules with CSV test case files.

Do not use this if you're working with already-supported providers (OpenAI, Anthropic, etc.) - stick with the built-in integrations. Avoid this approach if you need TypeScript-specific features, as this example uses plain JavaScript modules.

You provide a custom provider implementation in customProvider.mjs, a prompts file (prompts.txt), and test cases in CSV format (vars.csv). The example accepts standard promptfoo configuration including provider specifications and output destinations.

You receive output in JSON format as specified in the configuration.

Initialize the example:

npx promptfoo@latest init --example provider-custom/mjs
cd provider-custom/mjs

Run the evaluation:

promptfoo eval

For the full command-line equivalent with all options explicit:

promptfoo eval --prompts prompts.txt --tests vars.csv --providers openai:chat --output output.json --providers customProvider.js

promptfoo: The core evaluation framework that loads and executes the custom provider.

CSV test cases: Supports CSV file format for defining test variables and scenarios.

This example is for developers who need to implement custom providers for promptfoo. It requires JavaScript development skills to implement the provider interface.