Evaluate Prompt Performance with F-Score
A promptfoo example computing precision, recall, F1, and accuracy for GPT-4.1-mini IMDB sentiment classification.
Why it matters
Automate the evaluation of your AI prompts by calculating the F-score, a metric that combines precision and recall. This ensures a more robust and comprehensive assessment of prompt effectiveness.
Outcomes
What it gets done
Calculate F-score for prompt outputs.
Combine precision and recall metrics.
Evaluate prompt performance systematically.
Integrate into prompt testing pipelines.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-f-score | bash Overview
F Score
This promptfoo example classifies IMDB review sentiment with GPT-4.1-mini under a strict JSON schema, then computes precision, recall, F1, and accuracy via derivedMetrics. Use it when you need standard classification metrics beyond accuracy for a binary LLM classifier; it is scoped to positive/negative classification.
What it does
This promptfoo example evaluates openai:gpt-4.1-mini on binary sentiment classification of IMDB movie reviews, enforcing a strict JSON schema (sentiment, confidence, reasoning) via response_format, and then computes classic binary-classification metrics - precision, recall, F1 score, and accuracy - from the model's outputs against ground-truth labels.
When to use - and when NOT to
Use this example when you need proper classification metrics (not just pass/fail accuracy) for an LLM-based binary classifier - for example, to see whether a model is biased toward false positives or false negatives, not just its raw accuracy. It builds the metrics from four zero-weight javascript assertions (true_positives, false_positives, false_negatives, true_negatives) combined via derivedMetrics formulas, so it is specifically useful when promptfoo's built-in metrics aren't enough. It is scoped to binary (positive/negative) classification; multi-class problems would need the true/false-positive/negative logic reworked.
Inputs and outputs
The prompt asks the model to classify a review's sentiment and return JSON matching a json_schema response format (sentiment enum of positive/negative, confidence 1-10, reasoning string). defaultTest validates JSON shape with is-json, tracks raw accuracy, and computes the four confusion-matrix components with weight: 0 so they don't affect the pass/fail score directly:
derivedMetrics:
# Precision = TP / (TP + FP)
- name: precision
value: true_positives / (true_positives + false_positives)
# F1 Score = 2 * (precision * recall) / (precision + recall)
- name: f1_score
value: 2 * true_positives / (2 * true_positives + false_positives + false_negatives)
Test cases are loaded from file://imdb_eval_sample.csv rather than being defined inline.
Integrations
Runs against openai:gpt-4.1-mini with a structured json_schema response format, and uses promptfoo's javascript assertions with per-metric weight: 0 plus the derivedMetrics feature to compute precision, recall, F1, and accuracy from a CSV-loaded test set.
Who it's for
Teams building or evaluating LLM-based classifiers who need standard precision/recall/F1 metrics rather than simple accuracy, and want a promptfoo pattern for deriving those from raw prediction outcomes.
Source README
yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: IMDB Review Sentiment Analysis
providers:
- openai:gpt-4.1-mini
prompts:
label: Sentiment Analysis
raw: |
Analyze the sentiment of the following movie review. Classify it as either positive or negative.Review: "{{text}}"
Respond with a JSON object in the following format:
{
"sentiment": "positive" or "negative",
"confidence": number between 1-10,
"reasoning": "brief explanation"
}
config:
response_format:
type: json_schema
json_schema:
name: MovieReviewSentiment
schema:
type: object
properties:
sentiment:
type: string
enum: ['positive', 'negative']
confidence:
type: integer
minimum: 1
maximum: 10
reasoning:
type: string
required: ['sentiment', 'confidence', 'reasoning']
defaultTest:
assert:
# Basic JSON validation
- type: is-json
# Track binary classification metrics
- type: javascript
value: 'output.sentiment === context.vars.sentiment'
metric: accuracy
# For F1 score components (treating 'positive' as the positive class)
- type: javascript
value: "output.sentiment === 'positive' && context.vars.sentiment === 'positive' ? 1 : 0"
metric: true_positives
weight: 0
- type: javascript
value: "output.sentiment === 'positive' && context.vars.sentiment === 'negative' ? 1 : 0"
metric: false_positives
weight: 0
- type: javascript
value: "output.sentiment === 'negative' && context.vars.sentiment === 'positive' ? 1 : 0"
metric: false_negatives
weight: 0
- type: javascript
value: "output.sentiment === 'negative' && context.vars.sentiment === 'negative' ? 1 : 0"
metric: true_negatives
weight: 0
derivedMetrics:
Precision = TP / (TP + FP)
- name: precision
value: true_positives / (true_positives + false_positives)
Recall = TP / (TP + FN)
- name: recall
value: true_positives / (true_positives + false_negatives)
F1 Score = 2 * (precision * recall) / (precision + recall)
- name: f1_score
value: 2 * true_positives / (2 * true_positives + false_positives + false_negatives)
Accuracy = (TP + TN) / (TP + TN + FP + FN)
- name: accuracy_score
value: (true_positives + true_negatives) / (true_positives + true_negatives + false_positives + false_negatives)
tests: file://imdb_eval_sample.csv
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.