When should I use custom scoring overrides in promptfoo?

Use custom scoring overrides when promptfoo's default scoring behavior doesn't fit your use case and you need custom scoring logic. It's not needed if the default scoring already produces the pass/fail or score behavior you want.

What files do I need to set up custom scoring functions?

You need a `promptfooconfig.yaml` file plus JavaScript and/or Python files that export your custom scoring functions.

Prompt Chain

Automate Prompt Evaluation with Custom Assertions

Name: Eval Assertion Scoring Override
Availability: OnlineOnly
Author: Promptfoo

Three patterns for overriding promptfoo's default assertion scoring function in config, JavaScript, or Python.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Add to Favorites

Why it matters

Enhance your prompt engineering process by automating the evaluation of prompt outputs. This asset allows for custom assertion scoring, enabling more nuanced and precise quality control for your AI-generated content.

Outcomes

What it gets done

Define custom scoring logic for prompt evaluation assertions.

Integrate automated testing into your prompt development lifecycle.

Improve the reliability and consistency of AI model responses.

Debug and refine prompts based on detailed evaluation feedback.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-assertion-scoring-override | bash

Steps

Steps in the chain

Initialize the example project

Navigate to project directory

Run the evaluation

View the results

Overview

Eval Assertion Scoring Override

Demonstrates three ways to override promptfoo's default assertion scoring function: a global defaultTest config override, a named JavaScript export, or a Python function export. Use when the default scoring behavior doesn't fit and custom scoring logic is needed, in whichever of config/JS/Python fits the codebase.

What it does

This example demonstrates three different ways to define and override the default scoring function in promptfoo: a global override in the defaultTest section of the config, a named export in a JavaScript file, and a Python function export in a Python file.

When to use - and when NOT to

Use this when promptfoo's default scoring behavior doesn't fit your use case and you need custom scoring logic, choosing whichever of the three patterns (config-level, JS, or Python) best matches the rest of your codebase.

Not needed if the default scoring already produces the pass/fail or score behavior you want.

Inputs and outputs

Input is promptfooconfig.yaml plus JavaScript and/or Python files exporting the custom scoring functions. Output is a normal eval via promptfoo eval, viewable with promptfoo view.

Integrations

Custom scoring functions are plain JavaScript or Python code with no required external service integration.

Who it's for

promptfoo users who need scoring logic beyond the built-in default and want to choose between a config-level, JavaScript, or Python override.

npx promptfoo@latest init --example eval-assertion-scoring-override
cd eval-assertion-scoring-override

Source README

eval-assertion-scoring-override (Assertion Scoring Function Override Example)

You can run this example with:

npx promptfoo@latest init --example eval-assertion-scoring-override
cd eval-assertion-scoring-override

This example demonstrates different ways to define and override the default scoring function in promptfoo. It shows three patterns for implementing and referencing scoring functions:

A global override in the defaultTest section of the config
A named export in a JavaScript file
A Python function export in a Python file

Getting Started

Initialize the example:

npx promptfoo@latest init --example eval-assertion-scoring-override

Run the evaluation:

cd eval-assertion-scoring-override
promptfoo eval

View the results:

promptfoo view

FAQ

Common questions

Discussion