Load HuggingFace Datasets into LlamaIndex
Load HuggingFace datasets as LlamaIndex documents, with split, text-key, and streaming control.
Why it matters
Integrate HuggingFace Datasets with LlamaIndex to easily load and process large datasets for AI applications. This asset enables seamless data ingestion for retrieval-augmented generation (RAG) and other data-intensive tasks.
Outcomes
What it gets done
Load HuggingFace datasets as LlamaIndex documents.
Specify dataset splits, text keys, and cache directories.
Support lazy loading for streaming dataset samples.
Integrate with existing LlamaIndex pipelines for RAG.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-datasets | bash Overview
LlamaIndex Readers Integration: HuggingFace Datasets
A LlamaIndex reader that wraps datasets.load_dataset to load HuggingFace datasets as documents, with split and text-field control. Use for HuggingFace Hub-hosted datasets, with lazy loading for large datasets to avoid memory pressure.
What it does
The HuggingFace Datasets Reader loads HuggingFace datasets as LlamaIndex documents. DatasetsReader is instantiated with no required configuration, and load_data takes a dataset identifier (such as "lhoestq/demo1") plus optional arguments controlling exactly what gets loaded.
By default the train split is loaded, but a split argument (for example "test") selects a different one. A text_key argument names which dataset field should be used as the document's text value (for example "review"), and additional keyword arguments are passed straight through to the underlying datasets.load_dataset call - for example a cache_dir to control where the dataset is cached locally. A dataset that has already been loaded elsewhere can also be passed directly via a dataset argument, in which case all other loading arguments are ignored. For large datasets, lazy_load_data streams samples one at a time as an iterator, taking the same split, text_key, and a doc_id_key for naming which field becomes each document's ID.
When to use - and when NOT to
Use it when you want to bring a HuggingFace dataset - by name or from an already-loaded Dataset object - into LlamaIndex as documents, with control over which split and which field becomes the document text. Use lazy_load_data for large datasets where loading everything into memory upfront would be wasteful. Do not use it for data that isn't hosted on or compatible with the HuggingFace Hub; it is a thin wrapper over datasets.load_dataset, not a general-purpose file loader.
Capabilities
load_data/lazy_load_data load a named HuggingFace dataset (or an already-loaded one) as documents, with configurable split, text field, document ID field, and pass-through arguments to datasets.load_dataset such as cache_dir.
How to install
pip install llama-index-readers-datasets
Who it's for
Developers who need HuggingFace-hosted datasets loaded into LlamaIndex as documents, with control over split, text field, and memory use via streaming.
Source README
LlamaIndex Readers Integration: HuggingFace Datasets
Overview
HuggingFace Datasets Reader is a tool designed to load HuggingFace datasets as documents.
Installation
You can install HuggingFace Datasets Reader via pip:
pip install llama-index-readers-datasets
Usage
from llama_index.readers.datasets import DatasetsReader
from datasets import load_dataset
reader = DatasetsReader()
### Load train split (default) as metadata
docs = reader.load_data("lhoestq/demo1")
### Load test split as metadata
docs = reader.load_data("lhoestq/demo1", split="test")
### Load specify the dictionary key to use as text value
docs = reader.load_data("lhoestq/demo1", text_key="review")
### Pass additional arguments to datasets.load_dataset
docs = reader.load_data("lhoestq/demo1", cache_dir="/tmp/huggingface")
### Load from a preloaded dataset (ignore all other arguments)
dataset = load_dataset("lhoestq/demo1", split="train")
docs = reader.load_data(dataset=dataset)
### Lazy loading (stream samples)
for it in reader.lazy_load_data(
"lhoestq/demo1", split="test", text_key="review", doc_id_key="id"
):
print(it)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.