Securely Ingest Data with Pebblo for LlamaIndex
Wrap any LlamaIndex DocumentReader with Pebblo to surface compliance-relevant topics and entities in loaded data.
Maintainer of this project? Claim this page to edit the listing.
0.14.23Add to Favorites
Why it matters
Enhance your LlamaIndex RAG applications by integrating Pebblo's Safe DocumentReader. Ensure compliance and security by gaining deep visibility into the topics and entities within your ingested data before deployment.
Outcomes
What it gets done
Wrap existing Llama DocumentReaders with Pebblo Safe DocumentReader.
Identify and summarize semantic topics and entities in loaded data.
Enable safe data ingestion for Gen AI applications.
Generate reports or UI summaries of ingested data.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-pebblo | bash Overview
LlamaIndex Readers Integration: Pebblo
A LlamaIndex reader wrapper that adds Pebblo's topic and entity detection to an existing DocumentReader, surfacing what's actually being ingested into a Gen AI application. Use it to check ingested data for compliance-relevant topics and entities without replacing your existing document reader. Requires wrapping an existing reader such as CSVReader.
What it does
Pebblo Safe DocumentReader wraps an existing LlamaIndex DocumentReader so that data loaded into a Gen AI application is checked by Pebblo, which identifies semantic topics and entities present in the data and summarizes them in a UI or PDF report. Pebblo has two components: this Safe DocumentReader for LlamaIndex, and a separate Pebblo Daemon that the reader reports to.
When to use - and when NOT to
Use this reader when you need visibility into what kinds of topics and entities are actually being ingested into a RAG application - for example, to catch sensitive data before it reaches a deployed Gen AI app - without replacing your existing document loading code. It augments an existing reader rather than replacing it, so you still need a working LlamaIndex DocumentReader (such as CSVReader) to wrap.
Inputs and outputs
PebbloSafeReader is constructed around an existing reader instance, plus a mandatory app name and optional owner and description, and then used exactly like the reader it wraps:
from llama_index.readers.pebblo import PebbloSafeReader
from llama_index.readers.file import CSVReader
reader = CSVReader()
pebblo_reader = PebbloSafeReader(reader, name="acme-corp-rag-1",
owner="Joe Smith",
description="Support productivity RAG application")
documents = pebblo_reader.load_data(file=Path('data/corp_sens_data.csv'))
The call signature (load_data(file=...)) matches the underlying reader, so wrapping it requires only a few lines of change to existing code.
Integrations
Install with pip install llama-index-readers-pebblo, then wrap any existing LlamaIndex DocumentReader (the example uses CSVReader) with PebbloSafeReader. For the separate Pebblo Daemon component that this reader reports topic and entity data to, see Pebblo's own daemon documentation.
Who it's for
Teams building Gen AI or RAG applications who need compliance and security visibility into what data is being ingested, without rewriting their existing document-loading pipeline.
Source README
LlamaIndex Readers Integration: Pebblo
Pebblo Safe DocumentReader
Pebblo enables developers to safely load data and promote their Gen AI app to deployment without worrying about the organization’s compliance and security requirements. The project identifies semantic topics and entities found in the loaded data and summarizes them on the UI or a PDF report.
Pebblo has two components.
- Pebblo Safe DocumentReader for Llama
- Pebblo Daemon
This document describes how to augment your existing Llama DocumentReader with Pebblo Safe DocumentReader to get deep data visibility on the types of Topics and Entities ingested into the Gen-AI Llama application. For details on Pebblo Daemon see this pebblo daemon document.
Pebblo Safe DocumentReader enables safe data ingestion for Llama DocumentReader. This is done by wrapping the document reader call with Pebblo Safe DocumentReader
How to Pebblo enable Document Reading?
Assume a Llama RAG application snippet using CSVReader to read a CSV document for inference.
Here is the snippet of Document loading using CSVReader
from pathlib import Path
from llama_index.readers.file import CSVReader
reader = CSVReader()
documents = reader.load_data(file=Path('data/corp_sens_data.csv'))
print(documents)
The Pebblo SafeReader can be installed and enabled with few lines of code change to the above snippet.
Install PebbloSafeReader
pip install llama-index-readers-pebblo
Use PebbloSafeReader
from pathlib import Path
from llama_index.readers.pebblo import PebbloSafeReader
from llama_index.readers.file import CSVReader
reader = CSVReader()
pebblo_reader = PebbloSafeReader(reader, name="acme-corp-rag-1", # App name (Mandatory)
owner="Joe Smith", # Owner (Optional)
description="Support productivity RAG application")
documents = pebblo_reader.load_data(file=Path('data/corp_sens_data.csv'))
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.