Evaluate F-Score for LLM Outputs

Eval F Score is a promptfoo evaluation project that measures GPT-4o-mini's zero-shot performance on IMDB movie review sentiment analysis. It calculates F-score, precision, recall, and accuracy using promptfoo's assertion system, with each model response including sentiment classification, confidence scores (1-10), and reasoning.

Use this when you need to benchmark LLM sentiment classification performance against labeled datasets with rigorous statistical metrics. It's ideal for comparing model accuracy on binary sentiment tasks or establishing baseline performance before fine-tuning. Do not use this if you need multi-class classification beyond positive/negative sentiment, or if you require real-time evaluation rather than batch processing.

You provide an IMDB dataset sampled to 100 reviews (or customize the sample size), formatted as CSV with two columns: text (the movie review content) and sentiment (the label "positive" or "negative"). The evaluation tracks base metrics (true positives, false positives, false negatives) and calculates derived metrics including precision, recall, F1 score, and accuracy.

The project integrates with OpenAI's API (requires API key for GPT-4o-mini), HuggingFace's datasets library for IMDB data access, and promptfoo's evaluation framework. The dataset preparation script uses Python with dependencies managed via requirements.txt.

This is built for ML engineers and prompt engineers who need to validate LLM classification performance with standard information retrieval metrics. It's particularly useful for teams evaluating whether zero-shot prompting meets accuracy requirements before investing in fine-tuning, or for researchers comparing sentiment analysis approaches across different models.

To get started:

npx promptfoo@latest init --example eval-f-score
cd eval-f-score

Set your OpenAI API key and run:

promptfoo eval

The evaluation implements base metrics using JavaScript assertions:

- type: javascript
  value: "output.sentiment === 'positive' && context.vars.sentiment === 'positive' ? 1 : 0"
  metric: true_positives

Derived metrics are calculated after evaluation completes:

- name: precision
  value: true_positives / (true_positives + false_positives)

- name: f1_score
  value: 2 * true_positives / (2 * true_positives + false_positives + false_negatives)

To customize the dataset sample size, install Python dependencies and regenerate:

pip install -r requirements.txt
python prepare_data.py