Automate Assertion Scoring for Code Quality
Promptfoo eval config showing how to override the default score aggregation with a custom Python or JavaScript scoring function.
Why it matters
Enhance your code quality assurance by automating the process of assertion scoring. This asset helps ensure your code meets predefined quality standards through rigorous testing and verification.
Outcomes
What it gets done
Override default assertion scoring mechanisms.
Integrate custom assertion logic into your testing pipeline.
Improve code reliability through automated quality checks.
Streamline the code review process with objective scoring.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-assertion-scoring-override | bash Overview
Assertion Scoring Override
This Promptfoo example shows how to override the default geometric-mean score aggregation with a custom Python or JavaScript scoring function, either globally or per test case, on a translation-evaluation config. Use it as a template when Promptfoo's default scoring aggregation doesn't fit your evaluation and you need custom logic instead.
What it does
This Promptfoo example config demonstrates the assertScoringFunction option, which replaces Promptfoo's default per-test scoring aggregation (a geometric mean across assertion metrics) with a custom function written in Python or JavaScript - set either globally via defaultTest.assertScoringFunction or per test case. The example runs a translation task ({{language}}/{{text}} into a target language) against openai:gpt-4.1-mini, graded by three llm-rubric metrics (accuracy, fluency, grammar) plus a similar semantic-similarity assertion, with rubric grading and embeddings handled by a separate provider block (openai:gpt-4.1-mini for text, openai:embedding:text-embedding-3-small for embeddings). Three scoring modes are shown side by side: the default geometric mean, a Python override (override.py:calculate_score) for accuracy-only scoring, and a JavaScript override (override.js:overrideScoring) paired with a contains assertion.
When to use - and when NOT to
Use it as a starting template when Promptfoo's default geometric-mean aggregation across assertions doesn't fit your evaluation - for example, weighting one metric over others, or collapsing several assertions into a single pass/fail via a minimum-score rule. Do not use it if the default aggregation already fits, or if you're not running Promptfoo evals at all.
Inputs and outputs
Input: the YAML eval config itself, including providers, prompts, assertions, and either a default file://./default.js override or a per-test file://./override.py:calculate_score / file://./override.js:overrideScoring reference. Output: Promptfoo's evaluation report, with each test's score aggregated by whichever scoring function applies to it.
Integrations
Ties into Promptfoo's provider system (OpenAI chat and embedding models here) and its file-based scoring-function loader for Python and JavaScript.
Who it's for
Teams running Promptfoo evals who need custom score aggregation logic across multiple graded metrics, beyond the built-in default.
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.