Evaluate Model Factuality on HuggingFace Datasets

This Promptfoo config evaluates factual accuracy against the TruthfulQA benchmark. Tests aren't hand-written in the YAML - they're generated dynamically by a dataset_loader.ts:generate_tests script that pulls the EleutherAI/truthful_qa_mc dataset's validation split from HuggingFace and formats each question into Promptfoo test cases. The default provider is Anthropic's claude-sonnet-4-6, with commented-out alternatives for a Bedrock-hosted Llama model, OpenAI, and Google Gemini, so the config can be repointed to a different model or grading provider.

A defaultTest sets custom scoring weights for Promptfoo's factuality grading categories: an answer that's a strict subset of the correct answer scores 1.0, a superset scores 0.8, exact agreement scores 1.0, outright disagreement scores 0.0, and a factually compatible but differently worded answer scores 0.7.

Use it as a template for benchmarking a model's truthfulness against the TruthfulQA dataset, with fine-grained control over how partial or differently phrased but correct answers are scored. Do not use it if you need a custom or proprietary factuality dataset instead of TruthfulQA, or if the default equal-weight factuality scoring already fits your needs.

Input: the YAML config - prompt template, provider choice, and factuality scoring weights - plus a dataset_loader.ts script that fetches and formats the TruthfulQA validation split. Output: Promptfoo's evaluation report, scoring each generated test case against the weighted factuality categories.

Uses Promptfoo's factuality assertion and grading system with custom category weights, its dynamic test-generation via an external TypeScript loader, and the HuggingFace-hosted EleutherAI/truthful_qa_mc dataset.

Teams benchmarking a model's truthfulness on a standard dataset like TruthfulQA who need fine-grained control over how partially correct or differently worded answers get scored.