Validate AI Code Generation with External JavaScript Tests

Eval Javascript Assert External is a promptfoo example that demonstrates how to write custom JavaScript assertions in external files for evaluating LLM prompts. Instead of embedding test logic directly in your configuration, you define assertion functions in separate .js files that promptfoo loads and executes during evaluation runs.

Use this pattern when your assertion logic is complex enough to warrant separate files, when you need to reuse the same assertions across multiple promptfoo configurations, or when your team prefers to version-control test logic independently from configuration.

Do NOT use external JavaScript assertions if your test logic is simple enough to express inline in the YAML configuration - the added file management overhead isn't worth it for basic string matching or simple comparisons. Also avoid this approach if your team lacks JavaScript expertise, as debugging external assertion files requires more technical skill than working with built-in assertion types.

You provide an assert.js file containing your custom assertion functions and a promptfooconfig.yaml that references these external assertions. The configuration also specifies your test cases, prompts, and the LLM provider (such as OpenAI via the OPENAI_API_KEY environment variable).

After running the evaluation, you can view results through promptfoo's interface.

To initialize and run this example:

npx promptfoo@latest init --example eval-javascript-assert-external
cd eval-javascript-assert-external

Set your OPENAI_API_KEY environment variable, then edit assert.js and promptfooconfig.yaml to define your custom assertions and test cases. Execute the evaluation:

promptfoo eval

Afterwards, view results by running promptfoo view.

This example requires an OPENAI_API_KEY environment variable.

This example is designed for prompt engineers and QA teams who need programmatic control over their LLM test assertions.