What is an eval metric?

An eval metric is a measurement used in a promptfoo workflow to assess the performance of a model based on evaluation results. It can be customized with specific names and derived from existing metrics to provide tailored insights.

What are the metrics for named entity recognition evaluation?

The context does not specify the exact metrics for named entity recognition evaluation. It only mentions defining custom metric names and deriving new metrics from existing evaluation results.

What are the 12 important model evaluation metrics?

The context does not specify the 12 important model evaluation metrics. It only mentions defining custom metric names and deriving new metrics from existing evaluation results in a promptfoo workflow example.

In the context of a promptfoo workflow, 'eval' refers to the evaluation results that are used to define custom metric names and derive new metrics.

What are the four types of metrics?

The four types of metrics are custom metric names, derived metrics, existing evaluation results, and new metrics.

Prompt Chain

Customize Metric Evaluation in Promptfoo

Name: Eval Named Metrics
Availability: OnlineOnly
Author: Promptfoo

Create dynamic, static, and derived custom metric names in promptfoo assertions for more filterable eval results.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 19 days ago

Version 0.121.18

Add to Favorites

Why it matters

Enhance your prompt evaluation process by defining and utilizing custom named metrics within promptfoo. This asset allows for more nuanced and specific performance analysis of your AI models.

Outcomes

What it gets done

Define custom metrics for prompt evaluation.

Derive new metrics from existing evaluation results.

Integrate custom metrics into your promptfoo testing pipeline.

Analyze prompt performance with tailored evaluation criteria.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-named-metrics | bash

Steps

Steps in the chain

Setup eval-named-metrics example

Run evaluation

Overview

Eval Named Metrics

Demonstrates dynamic template-based metric names, static metric names, and derived formula metrics in promptfoo assertions, letting a single compact assertion definition still produce unique, filterable results per test case. Use when maintaining many similar test cases that need unique, filterable metric names generated from a single shared assertion definition.

What it does

This example demonstrates custom metric naming and derivation in promptfoo: dynamic metric names using template variables (e.g. {{speechStyle}}Style in defaultTest.assert) so a single assertion definition applies to all tests while still producing unique, filterable metric names per case; static, hardcoded metric names like Tone, Consistency, and Length; and derived metrics built from formulas on top of other metrics, such as DoubleConsistency = Consistency * 2. In the dynamic example, a test case with speechStyle: pirate produces a metric named pirateStyle, and one with speechStyle: nautical produces nauticalStyle, from the same defaultTest.assert definition.

When to use - and when NOT to

Use dynamic metric names when maintaining hundreds of similar test cases with a compact shared assertion definition, while still needing unique, filterable metric names per case. Use static names for simple, fixed metrics, and derived metrics when a useful score can be computed from an existing one via a formula.

Not needed for a small eval where a handful of plain, hardcoded metric names is already clear enough.

Inputs and outputs

Input is promptfooconfig.yaml with defaultTest.assert entries using template-variable metric names, static metric names, and derived-metric formulas. Output is a normal promptfoo eval run with results filterable by the resulting metric names.

Integrations

Uses promptfoo's built-in llm-rubric assertion type as the example grading mechanism; the metric-naming mechanism itself is provider-agnostic.

Who it's for

Teams maintaining large promptfoo test suites who need compact, reusable assertion definitions that still produce unique, filterable metric names.

npx promptfoo@latest init --example eval-named-metrics
cd eval-named-metrics

Source README

eval-named-metrics (Named Metrics Example)

This example demonstrates custom metric naming and derivation in promptfoo.

Setup

npx promptfoo@latest init --example eval-named-metrics
cd eval-named-metrics

Run

promptfoo eval

Features Demonstrated

Dynamic Metric Names: Using template variables like {{speechStyle}}Style in defaultTest.assert to apply different metric names per test case. This enables:
- A single assertion definition that applies to all tests
- Filtered results by the metric name that was dynamically generated
Static Metric Names: Hardcoded metric names like Tone, Consistency, and Length
Derived Metrics: Creating custom metrics based on formulas (e.g., DoubleConsistency = Consistency * 2)

Dynamic Metric Example

defaultTest:
  assert:
    - type: llm-rubric
      value: The response uses {{speechStyle}} speech patterns
      metric: '{{speechStyle}}Style'

tests:
  - vars:
      body: Ahoy there!
      speechStyle: pirate # Results in metric name "pirateStyle"

  - vars:
      body: Set sail!
      speechStyle: nautical # Results in metric name "nauticalStyle"

This allows you to maintain hundreds of test cases with a compact defaultTest configuration while still having unique, filterable metric names per test.

FAQ

Common questions

Discussion