What is Eval Rag and what does it demonstrate?

Eval Rag is a RAG (retrieval-augmented generation) evaluation example in promptfoo that demonstrates both successful and failing RAG metrics side by side. It includes intentionally unsupported context-recall facts to show what a mixed pass/fail RAG eval looks like.

What do I need to run this prompt chain?

You need to set the `OPENAI_API_KEY` environment variable. The input is a `promptfooconfig.yaml` file with context-recall assertions, and you run it using `npx promptfoo@latest eval`.

How do I get started with Eval Rag?

Run `npx promptfoo@latest init --example eval-rag` followed by `cd eval-rag` to initialize and enter the example directory.

Prompt Chain

Evaluate RAG Systems

Name: Eval Rag
Availability: OnlineOnly
Author: Promptfoo

A promptfoo RAG eval example that intentionally includes unsupported facts to show both passing and failing metrics.

Copy chain

Works with github

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Add to Favorites

Why it matters

Assess the performance and accuracy of Retrieval Augmented Generation (RAG) systems. This asset helps you systematically evaluate how well your RAG implementation retrieves relevant information and generates accurate responses.

Outcomes

What it gets done

Index data for RAG retrieval.

Query RAG system with test cases.

Summarize evaluation results.

Identify areas for RAG improvement.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-rag | bash

Steps

Steps in the chain

Set OPENAI_API_KEY environment variable

Edit promptfooconfig.yaml

Run evaluation

View results

Overview

Eval Rag

A promptfoo RAG eval example with context-recall assertions deliberately including unsupported facts, so results demonstrate both passing and failing RAG metrics. Use as a reference for context-recall assertion behavior, or as a starting template for a real RAG eval against your own retrieval data.

What it does

This is a RAG (retrieval-augmented generation) evaluation example in promptfoo. The config intentionally includes a couple of unsupported context-recall facts so the results show both successful and failing RAG metrics side by side; other model-graded scores can vary depending on the grading model used.

When to use - and when NOT to

Use this as a reference for what a mixed pass/fail RAG eval looks like in promptfoo, or as a template for building context-recall checks against your own retrieval system.

Not needed once you're building a real RAG eval against your own retrieval data rather than looking at this demonstration config.

Inputs and outputs

Requires OPENAI_API_KEY. Input is promptfooconfig.yaml with context-recall assertions, some intentionally unsupported. Output is npx promptfoo@latest eval, viewable with npx promptfoo@latest view.

Integrations

Runs against OpenAI via OPENAI_API_KEY; uses promptfoo's built-in context-recall RAG assertion type, with model-graded scores dependent on the grading model chosen.

Who it's for

Teams building or learning promptfoo's RAG/context-recall assertions who want a reference showing both a passing and a failing case.

npx promptfoo@latest init --example eval-rag
cd eval-rag

Source README

eval-rag (Rag Eval)

You can run this example with:

npx promptfoo@latest init --example eval-rag
cd eval-rag

Usage

To get started, set your OPENAI_API_KEY environment variable.

Next, edit promptfooconfig.yaml.

Then run:

npx promptfoo@latest eval

Afterwards, you can view the results by running npx promptfoo@latest view.

The config intentionally includes a couple of unsupported context-recall facts so the results show both successful and failing RAG metrics. Other model-graded scores can vary with the grading model.

FAQ

Common questions

Discussion