Evaluate prompts with custom JavaScript providers
Test AI prompts systematically with custom JavaScript providers and CSV test cases for quality assurance.
Why it matters
Test and validate AI prompts systematically by running them through custom provider implementations, enabling developers to ensure prompt quality and consistency before deployment.
Outcomes
What it gets done
Execute prompt evaluations using custom JavaScript provider modules
Process CSV test cases against multiple prompts automatically
Compare outputs from custom providers alongside standard LLM providers
Generate structured evaluation reports in JSON format
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-mjs | bash Overview
Mjs
A promptfoo example demonstrating custom provider implementation using JavaScript modules. Use when you need to create a custom provider for promptfoo evaluations.
What it does
This example shows how to use a custom API provider with promptfoo using a JavaScript module in .mjs format. It demonstrates the basic setup for running evaluations with a custom provider implementation and CSV test cases.
Inputs and outputs
The example uses a custom provider implementation in customProvider.mjs, prompt templates in prompts.txt, and test variables in a CSV file (vars.csv). The command-line example shows output being written to output.json.
Integrations
The command-line example shows the openai:chat provider being used alongside the custom provider, though the source does not detail the integration architecture.
Who it's for
This example is for developers who need to implement custom providers for promptfoo evaluations.
To get started, initialize the example:
npx promptfoo@latest init --example provider-custom/mjs
cd provider-custom/mjs
Run the evaluation:
promptfoo eval
The full command-line equivalent demonstrates the available configuration options:
promptfoo eval --prompts prompts.txt --tests vars.csv --providers openai:chat --output output.json --providers customProvider.js
You can adapt the example by modifying customProvider.mjs to implement your own provider logic.
Source README
provider-custom/mjs (Custom Provider Mjs)
You can run this example with:
npx promptfoo@latest init --example provider-custom/mjs
cd provider-custom/mjs
Usage
This example uses a custom API provider in customProvider.mjs. It also uses CSV test cases.
Run:
promptfoo eval
Full command-line equivalent:
promptfoo eval --prompts prompts.txt --tests vars.csv --providers openai:chat --output output.json --providers customProvider.js
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.