Evaluate LLM Outputs with G-Eval
Promptfoo config demonstrating the g-eval assertion for scoring LLM output on coherence, consistency, fluency, and relevance.
Why it matters
Automate the evaluation of language model outputs using a structured, multi-phase approach. This asset helps ensure the quality and accuracy of AI-generated content by comparing it against ground truth and predefined criteria.
Outcomes
What it gets done
Define evaluation criteria and ground truth.
Run LLM outputs through the G-Eval pipeline.
Classify and summarize evaluation results.
Review and analyze code quality based on LLM outputs.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-g-eval | bash Overview
G Eval
This Promptfoo example uses the g-eval assertion type to score LLM output on four G-Eval criteria - coherence, consistency, fluency, and relevance - both individually and combined into a single test. Use it as a template when scoring LLM output on established qualitative dimensions like coherence, consistency, fluency, and relevance.
What it does
This Promptfoo config demonstrates the g-eval assertion type, which implements the G-Eval methodology for LLM-based criteria scoring, derived from the DUC summarization-quality framework. Four criteria are defined as standalone tests against a simple greeting prompt run on openai:gpt-4.1-mini: Coherence (the reply should be well-structured, not just a heap of related information), Consistency (factual alignment with the source, penalizing hallucinated facts), Fluency (grammar, spelling, punctuation, word choice, sentence structure), and Relevance (only including source-important content, penalizing redundancy and excess information). A fifth test combines all four criteria into a single g-eval assertion that scores them together as a set rather than one at a time.
When to use - and when NOT to
Use it as a template for scoring LLM output on multiple qualitative dimensions - coherence, factual consistency, fluency, relevance - either individually or as a combined set. Do not use it if a single, simpler assertion type like llm-rubric for one custom criterion already covers your evaluation, or if your output isn't natural-language text these criteria apply to.
Inputs and outputs
Input: the YAML config - the prompt, provider, and the g-eval criteria definitions, either as single strings for individual tests or a list of strings for the combined test. Output: Promptfoo's evaluation report with a numeric score per g-eval criterion, or a combined score for the multi-criteria test.
Integrations
Uses Promptfoo's g-eval assertion type, backed here by OpenAI's gpt-4.1-mini as the grading provider.
Who it's for
Teams evaluating LLM output quality on established summarization-style criteria - coherence, consistency, fluency, relevance - who want either per-criterion scores or one combined evaluation.
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.