Customize Metric Naming and Derivation

This Promptfoo config demonstrates custom and dynamically named metrics for evaluation. A derivedMetrics block defines DoubleConsistency as a computed metric equal to Consistency * 2, combining an existing metric via a formula rather than a fresh assertion. A defaultTest shows dynamic metric naming: its llm-rubric assertion checks that a response uses {{speechStyle}} speech patterns, and its metric field itself contains a Nunjucks template variable ({{speechStyle}}Style), so the same single assertion definition renders to a differently named metric per test - "pirateStyle", "nauticalStyle", "archaicStyle" - letting results be filtered by style without duplicating the assertion. Two prompt templates rewrite input text as pirate speech or 17th-century seafarer speech.

Later tests layer static metrics alongside the dynamic one: a JSON-validation test tagged with the Consistency metric, which then feeds the derived DoubleConsistency metric, and a "complex validation" test with two more static metrics - Length (a JavaScript length check) and Relevance (a similar semantic check).

Use it as a template for organizing eval results under custom metric names, including metrics whose name is computed per test via Nunjucks templating, and metrics computed from other metrics via derivedMetrics. Do not use it if Promptfoo's default per-assertion-type reporting already gives you the breakdown you need.

Input: the YAML config - two style-rewriting prompt templates, a derivedMetrics formula, and test cases combining dynamic and static metric names. Output: Promptfoo's evaluation report broken down by named metric - pirateStyle, nauticalStyle, archaicStyle, Consistency, DoubleConsistency, Length, Relevance - rather than by raw assertion type alone.

Uses Promptfoo's metric field, including Nunjucks template variables for dynamic naming, and its derivedMetrics feature for computing one metric from another.

Teams that need eval results broken down by custom, meaningful metric names - especially when the same assertion logic needs to report under a different metric name per test case.