This is a promptfoo evaluation configuration benchmarking GPT-5 against GPT-5 Mini on the MMLU (Massive Multitask Language Understanding) academic benchmark, using an enhanced chain-of-thought prompt rather than a bare question. The shared prompt template instructs the model to act as "an expert test taker," present the question with its four multiple-choice options (A-D), think through the answer step by step, and conclude with a strictly formatted final line: "Therefore, the answer is A/B/C/D." Both providers are configured with max_tokens: 1000 to allow room for the requested reasoning trace before the final answer.

Its defaultTest assertion block applies three checks to every question: latency must stay under 60 seconds (a notably generous threshold reflecting that step-by-step reasoning takes longer than a bare answer), an llm-rubric model-graded assertion confirms the response actually includes clear step-by-step reasoning rather than just an answer, and a regex assertion enforces the exact required output format ("Therefore, the answer is [ABCD]") so automated grading can reliably extract the final choice. Test cases are pulled directly from HuggingFace's cais/mmlu dataset via promptfoo's huggingface:// dataset provider, starting deliberately with reasoning-heavy subjects - Abstract Algebra and Formal Logic, 10 questions each - chosen specifically because that's where improvements from the enhanced step-by-step prompting are expected to be most visible between the full and mini model.

Use this configuration when benchmarking GPT-5 against GPT-5 Mini, or as a template for comparing any two models, on structured academic multiple-choice reasoning, especially when evaluating whether chain-of-thought prompting narrows the capability gap between a full-size and mini model on logic-heavy subjects.

It is scoped to two specific MMLU subjects (Abstract Algebra, Formal Logic) at a sample size of 10 questions each - it is not a full MMLU run across all 57 subjects, so results here indicate a targeted, reasoning-heavy slice rather than comprehensive benchmark coverage.

Input is MMLU multiple-choice questions pulled live from HuggingFace's cais/mmlu dataset for the Abstract Algebra and Formal Logic subsets. Output is promptfoo's evaluation report comparing GPT-5 and GPT-5 Mini's accuracy (via the regex-extracted final answer), reasoning quality (via the llm-rubric check), and response latency for each question.

- huggingface://datasets/cais/mmlu?split=test&subset=abstract_algebra&config=abstract_algebra&limit=10

Built on the promptfoo evaluation framework, pulling test data directly from HuggingFace Datasets (cais/mmlu) via promptfoo's dataset-provider syntax, and targeting OpenAI's GPT-5 and GPT-5 Mini models with a shared chain-of-thought prompt template.

Developers or researchers comparing GPT-5 and GPT-5 Mini's academic reasoning performance, or building similar promptfoo evaluations that pull test cases directly from HuggingFace benchmark datasets with automated regex-based answer extraction.