What dataset does this prompt chain use for image classification evaluation?

It uses the Fashion MNIST dataset, with a CSV of 100 random Fashion MNIST images and their labels. A 10-sample CSV is included so you can skip the generation step, or you can regenerate it by running `python dataset_gen.py` after installing requirements.

Which AI models does this evaluation support?

It runs against OpenAI's GPT-4o and GPT-4o-mini by default, but can be adapted to other vision-capable models such as llama3.2 or Claude Sonnet 4.6 by changing the provider in the configuration.

What are the requirements to run this evaluation?

You need to set the `OPENAI_API_KEY` environment variable, install dependencies via `pip install -r requirements.txt`, and have the `promptfooconfig.yaml` file with the JSON schema and a sample-image CSV.

What does the structured JSON schema require for each image?

The schema requires classification, color analysis, and additional attributes for each image in the Fashion MNIST dataset.

Prompt Chain

Evaluate Image Classification Prompts

Name: Eval Image Classification
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example evaluating GPT-4o vision models on Fashion MNIST image classification with a structured schema.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 12 days ago

Version 0.121.19

Models

gpt 4o

Add to Favorites

Why it matters

This asset evaluates the performance of image classification prompts. It helps users understand how well their prompts are performing and identify areas for improvement in their AI models.

Outcomes

What it gets done

Run image classification prompts against a dataset.

Analyze and score the accuracy of classification results.

Provide insights into prompt effectiveness for image classification tasks.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-image-classification | bash

Steps

Steps in the chain

Set up your OpenAI API key

Run the evaluation

View the results

Re-generate or update the dataset

Experiment with the configuration

Overview

Eval Image Classification

A Promptfoo example evaluating GPT-4o and GPT-4o-mini on Fashion MNIST image classification using a structured JSON schema, with a dataset-generation script and swappable provider config. Use when evaluating vision-model image classification accuracy across providers or prompt and schema variations.

What it does

A Promptfoo example demonstrating image classification evaluation using the Fashion MNIST dataset, comparing GPT-4o and GPT-4o-mini through a structured JSON schema that captures classification, color analysis, and additional image attributes. A companion dataset_gen.py script can regenerate the dataset, pulling 100 random Fashion MNIST images and their labels into a CSV; a 10-image sample CSV ships with the example so that step can be skipped. The config is built to be experimented with directly: the JSON schema in promptfooconfig.yaml can be extended with new required fields, the provider can be swapped to a different model such as llama3.2 or Claude Sonnet 4.6, the system prompt can be tuned for classification accuracy, and additional assertions can be added to validate model outputs.

When to use - and when NOT to

Use as a starting reference for evaluating and comparing vision-capable models on a structured image-classification task with Promptfoo, or for testing how prompt and schema changes affect classification accuracy. Not the right example for text-only evaluation tasks, and not a production classification pipeline as-is - it's a Fashion MNIST demo meant to be adapted to a real dataset and schema.

Inputs and outputs

Input is an OPENAI_API_KEY environment variable and, optionally, a regenerated dataset CSV produced by dataset_gen.py, which needs its own pip-installed requirements. Output, after running the eval command, is Promptfoo's standard evaluation result set viewable via the view command, scored against the structured JSON schema's classification, color, and attribute fields.

npx promptfoo@latest init --example eval-image-classification
cd eval-image-classification

Integrations

Uses GPT-4o and GPT-4o-mini as the default compared models, swappable to others like llama3.2 or Claude Sonnet 4.6 via the provider config, the Fashion MNIST dataset, and a Python dataset-generation script with its own requirements.txt.

Who it's for

Developers evaluating vision-model image-classification accuracy across providers and prompt or schema variations who want a ready-to-run Promptfoo example rather than building the harness from scratch.

Source README

eval-image-classification (Image Classification Example with Promptfoo)

You can run this example with:

npx promptfoo@latest init --example eval-image-classification
cd eval-image-classification

This example demonstrates how to use Promptfoo for image classification tasks using the Fashion MNIST dataset. The example uses GPT-4o and GPT-4o-mini with a structured json schema to analyze images, including classification, color analysis, and additional attributes.

Getting Started

Set up your OpenAI API key:
```
export OPENAI_API_KEY='your-api-key'
```
Run the evaluation:
```
npx promptfoo@latest eval
```
View the results:
```
npx promptfoo@latest view
```
Optionally, re-generate or update the dataset:
```
python dataset_gen.py
```
Note: You may need to install dependencies with:
```
pip install -r requirements.txt
```
This script creates a CSV file with 100 random images from the Fashion MNIST dataset and their labels. A CSV with 10 sample images is included so you can skip this step if preferred.
Experiment with the configuration:
- Modify the JSON schema in promptfooconfig.yaml to add or adjust required fields
- Try different models such as llama3.2 or Claude Sonnet 4.6 by changing the provider in the config
- Adjust the system prompt to improve classification accuracy
- Add additional assertions to validate model outputs

FAQ

Common questions

Discussion