Extract Text and Data from PDFs
Convert PDFs to markdown for LlamaIndex using the GPL-3.0 licensed pdf-marker library.
Why it matters
Leverage the power of pdf-marker within LlamaIndex to efficiently extract text, tables, and images from PDF documents, preparing them for indexing and analysis.
Outcomes
What it gets done
Integrate pdf-marker for robust PDF content extraction.
Convert PDFs into structured markdown format.
Prepare extracted PDF data for RAG indexing.
Handle various document types, including scientific papers and books.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-pdf-marker | bash Overview
LlamaIndex Readers Integration: Pdf-Marker
A LlamaIndex reader that uses the GPL-3.0 licensed pdf-marker library to convert PDFs into markdown, preserving tables, equations, and images. Use for high-fidelity PDF-to-markdown conversion of books or scientific papers, after reviewing the GPL-3.0 license terms.
What it does
The Pdf-Marker Reader uses the pdf-marker library to extract the content of a PDF file for LlamaIndex, converting PDF to markdown quickly and accurately. Per the underlying marker project's own README, it is optimized for books and scientific papers, supports a wide range of documents and all languages, removes headers, footers, and other page artifacts, formats tables and code blocks, extracts and saves images alongside the markdown, converts most equations to LaTeX, and can run on GPU, CPU, or Apple's MPS.
PDFMarkerReader takes no special configuration - it is instantiated directly, and load_data is called with the path to the PDF file to convert.
When to use - and when NOT to
Use it when you need high-fidelity PDF-to-markdown conversion for LlamaIndex ingestion - particularly for books or scientific papers with tables, equations, and images that a simpler text-extraction approach would mangle. Do not use it in a commercial product without reviewing the license terms first: the underlying marker-pdf library is GPL-3.0 licensed, which requires any modifications or software incorporating the GPL-licensed code to also be made available under the GPL along with build and install instructions, and it carries additional commercial usage limitations documented in the marker project's own repository.
Capabilities
load_data converts a PDF file at a given path into markdown-formatted LlamaIndex documents, preserving tables, code blocks, and images, and converting most equations to LaTeX, via the underlying pdf-marker library.
How to install
from llama_index.readers.pdf_marker import PDFMarkerReader
from pathlib import Path
path = Path("/path/to/pdf")
reader = PDFMarkerReader()
reader.load_data(path)
The underlying marker-pdf library is GPL-3.0 licensed with additional commercial-use restrictions - review both before using it in a commercial product.
Who it's for
Developers who need accurate PDF-to-markdown conversion - preserving tables, equations, and images - for LlamaIndex ingestion, and who have reviewed the GPL-3.0 licensing terms of the underlying marker library.
Source README
LlamaIndex Readers Integration: Pdf-Marker
Uses the pdf-marker library to extract the content of a PDF file.
From the original README:
Marker converts PDF to markdown quickly and accurately.
- Supports a wide range of documents (optimized for books and scientific papers)
- Supports all languages
- Removes headers/footers/other artifacts
- Formats tables and code blocks
- Extracts and saves images along with the markdown
- Converts most equations to latex
- Works on GPU, CPU, or MPS
Usage
Here's an example usage of the PDFMarkerReader.
from llama_index.readers.pdf_marker import PDFMarkerReader
from pathlib import Path
path = Path("/path/to/pdf")
reader = PDFMarkerReader()
reader.load_data(path)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.