What does this RAG example do?

It demonstrates a complete RAG pipeline for asking questions over SEC filings using LangChain. It includes three components: `ingest.py` chunks and loads PDFs into a vector database, `retrieve.py` answers questions using that database, and `promptfooconfig.yaml` defines test inputs and requirements.

What setup is required to run this?

You need an `OPENAI_API_KEY`, a Python virtual environment (created with `python3 -m venv venv` and activated with `pip install -r requirements.txt`), and must run `python ingest.py` once to build the vector database. The Python virtual environment must stay active during evaluation, or `PROMPTFOO_PYTHON` must be set to the venv's Python path.

When should I use this instead of a simple prompt eval?

Use this as a template when you need to evaluate a full RAG pipeline including ingestion, retrieval, and answer generation—especially when comparing different retrieval configurations. It's not needed for simple prompt evals without retrieval, or if you already have your own ingestion/retrieval code.

What integrations does this use?

It uses LangChain for PDF chunking and vector-database retrieval, OpenAI for question answering, and pulls source PDFs from a public Google Cloud Storage bucket.

Prompt Chain

Evaluate RAG System Performance

Name: Eval Rag Full
Availability: OnlineOnly
Author: Promptfoo

A full promptfoo RAG example over SEC filings, with a PDF ingest pipeline and a Python retrieval provider.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 27 days ago

Version 0.121.18

Models

universal

Add to Favorites

Why it matters

Assess the effectiveness of your Retrieval Augmented Generation (RAG) system by running comprehensive evaluations. This asset helps you test and refine your RAG pipeline to ensure accurate and relevant information retrieval and generation.

Outcomes

What it gets done

Index knowledge base for RAG.

Query RAG system with test cases.

Summarize evaluation results.

Write tests for RAG components.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-rag-full | bash

Steps

Steps in the chain

Set OPENAI_API_KEY environment variable

Create Python virtual environment

Activate virtual environment

Install Python dependencies

Run ingest.py to create vector database

Edit promptfooconfig.yaml

Edit retrieve.py

Run promptfoo eval

View results

Overview

Eval Rag Full

This promptfoo example evaluates a full RAG pipeline over SEC filings, using ingest.py to build a vector database, a Python retrieve.py provider to answer questions, and a config that can compare two retrieval setups. Use it as a template for evaluating a RAG pipeline end-to-end with promptfoo, including the document ingest step.

What it does

This RAG example allows you to ask questions over a number of public company SEC filings, using LangChain, though the flow is representative of any RAG solution. It has three parts: ingest.py chunks and loads PDFs (pulled from a public Google Cloud bucket) into a vector database, retrieve.py is a promptfoo-compatible provider that answers RAG questions using that database, and promptfooconfig.yaml holds the test inputs and requirements.

npx promptfoo@latest init --example eval-rag-full
cd eval-rag-full

When to use - and when NOT to

Use this as a template for evaluating a RAG pipeline end-to-end with promptfoo, including the ingest step, not just the retrieval/answer step. Since promptfoo is a Node.js CLI but the file://retrieve.py provider runs inside Python, you must keep the Python virtual environment active when running the eval, or set PROMPTFOO_PYTHON=./venv/bin/python so promptfoo can import the packages from requirements.txt.

Inputs and outputs

Setup: set OPENAI_API_KEY, create and activate a Python virtual environment (python3 -m venv venv, source venv/bin/activate), install dependencies (pip install -r requirements.txt), then run python ingest.py to build the vector database. Edit promptfooconfig.yaml to configure the questions you want to test and retrieve.py to control how context is loaded and questions are answered, then run npx promptfoo@latest eval and view results with npx promptfoo@latest view. A second config, promptfooconfig.with-asserts.yaml, is a more complete example that compares two RAG configurations side by side - the smaller retrieval configuration is intentionally set up to miss a couple of details, so the comparison view demonstrates both failures and passes.

Who it's for

Teams building a RAG system over documents like SEC filings who want an end-to-end evaluation template - ingest, retrieval provider, and test config - to measure and compare retrieval configurations rather than evaluating prompts in isolation.

Source README

eval-rag-full (Rag Full)

You can run this example with:

npx promptfoo@latest init --example eval-rag-full
cd eval-rag-full

Usage

This RAG example allows you to ask questions over a number of public company SEC filings. It uses LangChain, but the flow is representative of any RAG solution.

There are 3 parts:

ingest.py: Chunks and loads PDFs into a vector database (PDFs are pulled from a public Google Cloud bucket)
retrieve.py: Promptfoo-compatible provider that answers RAG questions using the database.
promptfooconfig.yaml: Test inputs and requirements.

To get started:

Set the OPENAI_API_KEY environment variable.
Create a python virtual environment: python3 -m venv venv
Enter the environment: source venv/bin/activate
Install python dependencies: pip install -r requirements.txt
Run ingest.py to create the vector database: python ingest.py

Now we're ready to go.

Edit promptfooconfig.yaml to your liking to configure the questions you'd like to ask in your tests. Then run:
Edit retrieve.py to control how context is loaded and questions are answered.

npx promptfoo@latest eval

Promptfoo is a Node.js CLI, but the file://retrieve.py provider runs inside Python. Keep the virtual environment active when running the eval, or set PROMPTFOO_PYTHON=./venv/bin/python so Promptfoo can import the packages from requirements.txt.

Afterwards, you can view the results by running npx promptfoo@latest view

See promptfooconfig.with-asserts.yaml for a more complete example that compares the performance of two RAG configurations. The smaller retrieval configuration is intentionally expected to miss a couple of details so the comparison view demonstrates failures as well as passes.

FAQ

Common questions

Discussion