Load and Chunk PDFs for LLM Analysis
Smart PDF Loader is a fast PDF reader for LlamaIndex that parses layout structure (sections, lists, tables) and chunks content optimally for LLM retrieval.
Why it matters
Ingest and intelligently chunk PDF documents, preserving structural information like sections, lists, and tables, to optimize them for large language model processing and retrieval.
Outcomes
What it gets done
Parse PDF content with layout awareness.
Create optimal context chunks for LLMs.
Enable retrieval augmented generation (RAG) on PDF data.
Extract structured information from PDFs.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-smart-pdf-loader | bash Overview
Smart PDF Loader
Smart PDF Loader is a fast PDF reader for LlamaIndex that parses document layout structure including nested sections, lists, paragraphs, and tables. It chunks PDF content into optimal short contexts for large language models. Use Smart PDF Loader when building retrieval-augmented generation systems that need to process PDFs with tables, hierarchical sections, or nested lists. The source material demonstrates its use for querying documents with VectorStoreIndex and query engines.
What it does
Smart PDF Loader is a PDF reader for LlamaIndex that understands document layout structure including nested sections, nested lists, paragraphs, and tables. It uses this layout information to chunk PDFs into optimal short contexts for large language models.
Inputs and outputs
You provide either a PDF URL (e.g., an arXiv paper link) or a local file path to a PDF document, along with the llmsherpa API endpoint URL. The loader returns documents that can be used with LlamaIndex components.
Integrations
Smart PDF Loader works with LlamaIndex components as shown in the source examples, including VectorStoreIndex and query engines. It is based on LayoutPDFReader from the llmsherpa library.
Who it's for
This tool is for users building applications with LlamaIndex who need to process PDF documents while preserving layout structure.
Installation and usage
Install the reader and its dependency:
pip install llama-index-readers-smart-pdf-loader
pip install llmsherpa
Basic usage example:
from llama_index.readers.smart_pdf_loader import SmartPDFLoader
llmsherpa_api_url = "https://readers.llmsherpa.com/api/document/developer/parseDocument?renderFormat=all"
pdf_url = "https://arxiv.org/pdf/1910.13461.pdf" # also allowed is a file path e.g. /home/downloads/xyz.pdf
pdf_loader = SmartPDFLoader(llmsherpa_api_url=llmsherpa_api_url)
documents = pdf_loader.load_data(pdf_url)
For retrieval-augmented generation:
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("list all the tasks that work with bart")
print(response)
Source README
Smart PDF Loader
pip install llama-index-readers-smart-pdf-loader
SmartPDFLoader is a super fast PDF reader that understands the layout structure of PDFs such as nested sections, nested lists, paragraphs and tables.
It uses layout information to smartly chunk PDFs into optimal short contexts for LLMs.
Requirements
Install the llmsherpa library if it is not already present:
pip install llmsherpa
Usage
Here's an example usage of the SmartPDFLoader:
from llama_index.readers.smart_pdf_loader import SmartPDFLoader
llmsherpa_api_url = "https://readers.llmsherpa.com/api/document/developer/parseDocument?renderFormat=all"
pdf_url = "https://arxiv.org/pdf/1910.13461.pdf" # also allowed is a file path e.g. /home/downloads/xyz.pdf
pdf_loader = SmartPDFLoader(llmsherpa_api_url=llmsherpa_api_url)
documents = pdf_loader.load_data(pdf_url)
Now you can use the documents with other LlamaIndex components. For example, for retrieval augmented generation, try this:
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("list all the tasks that work with bart")
print(response)
response = query_engine.query("what is the bart performance score on squad")
print(response)
More Examples
SmartPDFLoader is based on LayoutPDFReader from llmsherpa library. See the documentation there to explore other ways to use the library for connecting data from your PDFs with LLMs.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.