What is G-Eval and when should I use it?

G-Eval is an LLM-graded evaluation method (based on arxiv 2303.16634) for scoring model output quality. Use it when you need LLM-based grading; if a simpler deterministic assertion like string matching or regex already captures what you need to check, G-Eval is not necessary.

How do I set up and run this G-Eval example?

Run `npx promptfoo@latest init --example eval-g-eval`, then navigate to the `eval-g-eval` directory and execute `promptfoo eval` to run the evaluation.

What configuration file does G-Eval use?

G-Eval is configured in `promptfooconfig.yaml` as an LLM-graded assertion that relies on a grading LLM provider to score outputs.

Who is this prompt chain intended for?

This is designed for promptfoo users who want to apply the G-Eval LLM-based grading method to evaluate and score output quality.

Prompt Chain

Evaluate and Refine AI Model Outputs

Name: Eval G Eval
Availability: OnlineOnly
Author: Promptfoo

A promptfoo example demonstrating G-Eval usage, referencing the research paper arxiv.org/abs/2303.16634.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 3 days ago

Version 0.121.19

Add to Favorites

Why it matters

Automate the evaluation of AI model outputs against a set of criteria, providing structured feedback for iterative improvement.

Outcomes

What it gets done

Define evaluation metrics and criteria.

Run AI model outputs through evaluation prompts.

Analyze and summarize evaluation results.

Generate refined outputs or code based on feedback.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-g-eval | bash

Overview

Eval G Eval

This example shows how you can use G-Eval with promptfoo. G-Eval is referenced in the research paper available at arxiv.org/abs/2303.16634. Use this example when you want to see how G-Eval can be used with promptfoo. Do not use this if you need evaluation methods other than G-Eval, or if you're looking for a standalone G-Eval implementation outside the promptfoo toolchain.

What it does

This example shows how you can use G-Eval with promptfoo. G-Eval is referenced in the research paper available at arxiv.org/abs/2303.16634.

When to use - and when NOT to

Use this example when you want to see how G-Eval can be used with promptfoo.

Do not use this if you need evaluation methods other than G-Eval, or if you're looking for a standalone G-Eval implementation outside the promptfoo toolchain.

Inputs and outputs

You configure the example through promptfooconfig.yaml.

Integrations

This example works within promptfoo's workflow and references G-Eval from the research paper.

Who it's for

This example is available for anyone using promptfoo who wants to explore G-Eval.

To get started, initialize the example:

npx promptfoo@latest init --example eval-g-eval
cd eval-g-eval

Then run the evaluation:

promptfoo eval

The example is configured through promptfooconfig.yaml.

Source README

eval-g-eval (G Eval)

You can run this example with:

npx promptfoo@latest init --example eval-g-eval
cd eval-g-eval

Usage

This example shows how you can use G-Eval with promptfoo

Configure in promptfooconfig.yaml. Run with:

promptfoo eval

FAQ

Common questions

Discussion