Implement Self-RAG for Enhanced Document Analysis
An archived LangGraph example implementing Self-RAG's self-grading of retrieval, relevance, and usefulness.
Why it matters
Enhance retrieval-augmented generation (RAG) systems by incorporating self-reflection and self-grading mechanisms. This asset allows for more accurate and relevant information retrieval and generation by evaluating the quality and relevance of retrieved documents and the LLM's responses.
Outcomes
What it gets done
Implement a self-reflection loop for document retrieval decisions.
Evaluate the relevance of retrieved passages to user queries.
Assess LLM generation for factual consistency with retrieved documents.
Grade the overall usefulness of generated responses.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/lg-langgraphselfrag | bash Steps
Steps in the chain
Overview
Self-RAG
An archived LangGraph example implementing Self-RAG's four self-grading decisions for retrieval, relevance, and usefulness. Use it as a reference pattern for RAG that self-grades retrieval necessity, relevance, and answer usefulness at each stage.
What it does
An archived LangGraph example (no longer maintained - see LangChain's consolidated documentation for current guidance) implementing Self-RAG: a RAG strategy built around four self-reflection decisions from the Self-RAG paper. It decides (1) whether retrieval is even needed for the question, (2) whether each retrieved chunk is relevant to the question, (3) whether the generation from each chunk is fully, partially, or not supported by that chunk (checking for hallucination), and (4) how useful the generation is as a response to the question, scored 1-5.
When to use - and when NOT to
Use this as a reference implementation for a RAG pipeline that self-grades at every stage - retrieval necessity, document relevance, groundedness of the generation, and answer usefulness - rather than a single-pass retrieve-and-generate flow. It is explicitly archived and no longer updated, so treat it as a study pattern rather than a production dependency - check LangChain's current consolidated documentation for up-to-date APIs.
Inputs and outputs
Decision 1 takes a question (and optionally a prior generation) and outputs yes/no/continue on whether to retrieve. Decision 2 takes the question plus each retrieved chunk and outputs relevant/irrelevant. Decision 3 takes the question, chunk, and generation, and outputs fully supported/partially supported/no support. Decision 4 takes the question and generation and scores usefulness 1-5. The example indexes three blog posts as its retrieval source and builds the full decision flow as a LangGraph graph.
Integrations
Built with LangGraph and LangChain, with LangSmith recommended for tracing and debugging - the source links two public LangSmith trace examples showing the graph's actual run-time behavior.
Who it's for
Developers building a RAG pipeline that needs explicit self-grading at each stage - whether to retrieve, whether retrieved content is relevant, whether the generation is grounded, and whether the answer is actually useful - instead of trusting a single retrieve-and-generate pass.
Source README
This directory is retained purely for archival purposes and is no longer updated. Please see the newly consolidated LangChain documentation for the most current information and resources.
Self-RAG
Self-RAG is a strategy for RAG that incorporates self-reflection / self-grading on retrieved documents and generations.
In the paper, a few decisions are made:
- Should I retrieve from retriever,
R-
- Input:
x (question)ORx (question),y (generation) - Decides when to retrieve
Dchunks withR - Output:
yes, no, continue
- Are the retrieved passages
Drelevant to the questionx-
- Input: (
x (question),d (chunk)) fordinD
- Input: (
dprovides useful information to solvex- Output:
relevant, irrelevant
- Are the LLM generation from each chunk in
Dis relevant to the chunk (hallucinations, etc) -
- Input:
x (question),d (chunk),y (generation)fordinD - All of the verification-worthy statements in
y (generation)are supported byd - Output:
{fully supported, partially supported, no support
- The LLM generation from each chunk in
Dis a useful response tox (question)-
- Input:
x (question),y (generation)fordinD y (generation)is a useful response tox (question).- Output:
{5, 4, 3, 2, 1}
We will implement some of these ideas from scratch using LangGraph.
Setup
First let's install our required packages and set our API keys
Set up LangSmith for LangGraph development
Sign up for LangSmith to quickly spot issues and improve the performance of your LangGraph projects. LangSmith lets you use trace data to debug, test, and monitor your LLM apps built with LangGraph - read more about how to get started here.
Retriever
Let's index 3 blog posts.
LLMs
Graph
Capture the flow in as a graph.
Graph state
Build Graph
The just follows the flow we outlined in the figure above.
LangSmith Traces -
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.