What model and task does this evaluation use?

This project evaluates GPT-4o-mini's zero-shot performance on IMDB movie review sentiment analysis, classifying reviews as positive or negative sentiment.

What metrics does this project calculate?

It calculates precision, recall, F1 score, and accuracy by first computing base metrics (true/false positives and negatives) per test case using JavaScript assertions, then deriving the final metrics from those base values.

What setup is required to run this project?

You need to set the `OPENAI_API_KEY` environment variable, install dependencies with `pip install -r requirements.txt`, and optionally run `python prepare_data.py` to regenerate the dataset with a different sample size.

Prompt Chain

Evaluate F-Score for LLM Outputs

Name: Eval F Score
Availability: OnlineOnly
Author: Promptfoo

Evaluate GPT-4o-mini's zero-shot IMDB sentiment classification with precision, recall, and F1 metrics in promptfoo.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

gpt 4o

Add to Favorites

Why it matters

Automate the evaluation of language model outputs using the F-score metric. This asset helps ensure the quality and relevance of generated text by comparing it against ground truth.

Outcomes

What it gets done

Calculate precision and recall for LLM responses.

Compute the F-score to balance precision and recall.

Integrate F-score evaluation into your LLM testing pipelines.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-f-score | bash

Steps

Steps in the chain

Set OpenAI API key and run evaluation

Install Python dependencies

Prepare dataset

Calculate base metrics

Calculate derived metrics

Overview

Eval F Score

Evaluates GPT-4o-mini's zero-shot IMDB sentiment classification in promptfoo, computing true/false positive and negative counts via JavaScript assertions and deriving precision, recall, F1, and accuracy after the run. Use when a classification-style eval needs standard precision/recall/F1/accuracy metrics instead of a simple pass rate.

What it does

This project evaluates GPT-4o-mini's zero-shot performance on IMDB movie review sentiment analysis using promptfoo. Each model response includes a sentiment classification, a confidence score (1-10), and reasoning. The dataset comes from HuggingFace's datasets library, sampled to 100 reviews and preprocessed into a CSV with text and sentiment columns (prepare_data.py regenerates it with a different sample size, requiring pip install -r requirements.txt). Base metrics (true/false positives and negatives) are computed per test case with JavaScript assertions, e.g. output.sentiment === 'positive' && context.vars.sentiment === 'positive' ? 1 : 0 tagged with metric: true_positives. Derived metrics are then calculated from those base metrics after the run completes: precision (TP/(TP+FP)), recall (TP/(TP+FN)), F1 score (2*(precision*recall)/(precision+recall)), and accuracy ((TP+TN)/Total).

When to use - and when NOT to

Use this pattern when you need standard classification metrics (precision, recall, F1, accuracy) out of an LLM eval rather than a simple pass rate, particularly for a binary or categorical classification task with a labeled dataset.

Not needed for open-ended generation tasks with no ground-truth label to compare against, where classification metrics don't apply.

Inputs and outputs

Requires OPENAI_API_KEY. Setup for regenerating the dataset needs pip install -r requirements.txt then python prepare_data.py. Inputs are promptfooconfig.yaml with the JavaScript base-metric assertions and derived-metric formulas. Output is promptfoo eval, reporting per-model precision/recall/F1/accuracy.

Integrations

Sources its evaluation dataset from HuggingFace's datasets library (IMDB) and runs the classification model via OpenAI (OPENAI_API_KEY).

Who it's for

Teams evaluating a classification-style LLM task who need standard precision/recall/F1/accuracy metrics rather than a simple pass rate.

npx promptfoo@latest init --example eval-f-score
cd eval-f-score

Source README

eval-f-score (F-Score HuggingFace Dataset Sentiment Analysis Eval)

You can run this example with:

npx promptfoo@latest init --example eval-f-score
cd eval-f-score

This project evaluates GPT-4o-mini's zero-shot performance on IMDB movie review sentiment analysis using promptfoo. Each model response includes:

Sentiment classification
Confidence score (1-10)
Reasoning for the classification

Quick Start

Set your OpenAI API key and run the evaluation:

promptfoo eval

Dataset

The evaluation uses the IMDB dataset from HuggingFace's datasets library, sampled to 100 reviews. The dataset is preprocessed into a CSV with two columns:

text: The movie review content
sentiment: The label ("positive" or "negative")

To modify the sample size or generate a new dataset, you can use prepare_data.py. First, install the Python dependencies:

pip install -r requirements.txt

Then run the preparation script:

python prepare_data.py

Metrics Overview

The evaluation implements F-score and related metrics using promptfoo's assertion system:

Base Metrics calculated for each test case using JavaScript assertions:

- type: javascript
  value: "output.sentiment === 'positive' && context.vars.sentiment === 'positive' ? 1 : 0"
  metric: true_positives

Derived Metrics calculated from base metrics after the evaluation completes:

- name: precision
  value: true_positives / (true_positives + false_positives)

- name: f1_score
  value: 2 * true_positives / (2 * true_positives + false_positives + false_negatives)

The evaluation tracks:

True/False Positives/Negatives: Base metrics for classification
Precision: TP / (TP + FP)
Recall: TP / (TP + FN)
F1 Score: 2 × (precision × recall) / (precision + recall)
Accuracy: (TP + TN) / Total

FAQ

Common questions

Discussion