What does the Eval Self Grading prompt chain do?

It demonstrates how to have an LLM grade its own output according to predefined expectations within promptfoo, using configuration defined in `promptfooconfig.yaml` or a separate CSV file.

When should I use self-grading versus other grading methods?

Use self-grading when you want the same model to assess its own output against stated expectations. Use independent grading models or deterministic assertions instead when grading should come from a separate evaluator.

What are the input and output formats for this prompt chain?

Input is either `promptfooconfig.yaml` or `tests.csv` (when using `promptfoo eval --tests tests.csv`), and the output is a standard `promptfoo eval` run result.

How do I get started with this example?

Run `npx promptfoo@latest init --example eval-self-grading`, then navigate into the created directory with `cd eval-self-grading`.

Prompt Chain

Automate Code Evaluation with Self-Grading

Name: Eval Self Grading
Availability: OnlineOnly
Author: Promptfoo

A promptfoo example that demonstrates how to have an LLM grade its own output according to predefined expectations, with configuration via YAML or CSV test

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 3 days ago

Version 0.121.19

Models

gpt 4o

Add to Favorites

Why it matters

Streamline your code quality assurance by automating the evaluation of code against predefined criteria. This prompt chain enables self-grading of code, reducing manual review time and ensuring consistent quality.

Outcomes

What it gets done

Define evaluation criteria for code quality.

Automatically grade code snippets based on defined criteria.

Identify areas for code improvement through automated feedback.

Integrate self-grading into CI/CD pipelines for continuous quality checks.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-self-grading | bash

Overview

Eval Self Grading

A promptfoo example demonstrating self-grading functionality where an LLM evaluates its own outputs against predefined expectations. This example is provided as a reference implementation for developers exploring self-grading evaluation patterns with promptfoo.

What it does

Eval Self Grading is a promptfoo example that shows how you can have an LLM grade its own output according to predefined expectations. The configuration is provided in promptfooconfig.yaml, and you can also define tests in a CSV file.

Inputs and outputs

You provide a promptfooconfig.yaml file for configuration. Alternatively, you can supply test definitions in a CSV file format using the --tests flag.

How to use

Initialize and run the example:

npx promptfoo@latest init --example eval-self-grading
cd eval-self-grading

Run the evaluation with the default YAML configuration:

promptfoo eval

Or run with tests defined in a CSV file:

promptfoo eval --tests tests.csv

Source README

eval-self-grading (Self Grading)

You can run this example with:

npx promptfoo@latest init --example eval-self-grading
cd eval-self-grading

Usage

This example shows how you can have an LLM grade its own output according to predefined expectations.

The configuration is provided in promptfooconfig.yaml.

Run:

promptfoo eval

You can also define the tests in a CSV file:

promptfoo eval --tests tests.csv

FAQ

Common questions

Discussion