Evaluate LLM Output Quality with BERTScore Similarity
Measures semantic similarity between LLM outputs and reference text using BERTScore in a promptfoo assertion.
Why it matters
Automatically measure semantic similarity between LLM-generated outputs and reference text using BERTScore, enabling objective quality assessment and regression testing of language model responses with configurable thresholds.
Outcomes
What it gets done
Compare LLM outputs against reference text using BERT embeddings
Score semantic similarity on a 0-1 scale with configurable pass thresholds
Test against multiple valid reference answers and select best match
Integrate BERTScore assertions into automated evaluation pipelines
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-eval-bert-score | bash Steps
Steps in the chain
Overview
Eval Bert Score
Uses BERTScore to measure semantic similarity between LLM outputs and one or more reference texts as a promptfoo python assertion. Use when evaluating LLM output quality by semantic similarity rather than exact string matching.
What it does
This promptfoo example uses BERTScore to measure semantic similarity between LLM outputs and reference text, via a Python assertion:
tests:
- vars:
text: 'Hello world'
reference: 'Hi there'
assert:
- type: python
value: file://bertscore_check.py
threshold: 0.7 # Pass if similarity > 70%
An advanced variant compares an output against multiple valid references using the bert_score library directly, computing an F1 score against each candidate reference and returning the best (maximum) match. BERTScore returns a similarity score from 0 to 1: 0.9 or higher means nearly identical meaning, 0.7-0.9 means similar meaning, and below 0.7 means different meaning.
When to use - and when NOT to
Use this example when evaluating LLM output quality by semantic similarity to a reference answer, including cases with multiple acceptable reference answers, rather than exact string matching.
Inputs and outputs
Requires pip install -r requirements.txt; the first run downloads the BERT model (~1.4GB). Run the basic example via promptfoo eval, or the advanced multi-reference example via promptfoo eval -c promptfooconfig-advanced.yaml.
Integrations
Built on the bert_score Python library, invoked as a promptfoo python-type assertion with a configurable pass threshold.
Who it's for
Engineers evaluating LLM outputs for semantic correctness against one or more reference answers who need a similarity score more tolerant of paraphrasing than exact-match or simple string-similarity checks.
Source README
eval-bert-score (BERTScore Evaluation)
Use BERTScore to measure semantic similarity between LLM outputs and reference text.
npx promptfoo@latest init --example eval-bert-score
cd eval-bert-score
Setup
pip install -r requirements.txt
Note: First run will download the BERT model (~1.4GB).
Usage
Basic Example
### promptfooconfig.yaml
tests:
- vars:
text: 'Hello world'
reference: 'Hi there'
assert:
- type: python
value: file://bertscore_check.py
threshold: 0.7 # Pass if similarity > 70%
Run: promptfoo eval
Advanced Example
Compare against multiple valid references:
### promptfooconfig-advanced.yaml
assert:
- type: python
value: |
from bert_score import score
references = [
"First valid answer",
"Second valid answer",
"Third valid answer"
]
scores = []
for ref in references:
_, _, F1 = score([output], [ref], lang='en', verbose=False)
scores.append(F1.item())
return max(scores) # Use best match
Run: promptfoo eval -c promptfooconfig-advanced.yaml
How It Works
BERTScore returns a similarity score from 0 to 1:
- 0.9+ = Nearly identical meaning
- 0.7-0.9 = Similar meaning
- <0.7 = Different meaning
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.