Load Hugging Face datasets into LlamaIndex pipelines

Hugging Face FS Loader is a data reader for LlamaIndex that uses Hugging Face Hub's Filesystem API (version 0.14 or higher) to load datasets. It provides three flexible loading methods: load_data for documents, load_dicts for dictionary objects, and load_df for pandas DataFrames, allowing you to choose the format that best suits your downstream processing needs.

Use this loader when you need to ingest datasets hosted on Hugging Face Hub directly into LlamaIndex workflows. It's ideal for scenarios where you're building RAG applications, indexing public datasets for search, or processing machine learning datasets within LlamaIndex pipelines. The multiple output formats make it suitable whether you need structured documents for indexing, raw dictionaries for custom processing, or DataFrames for analytical operations.

Do NOT use this loader if you need to access datasets from sources other than Hugging Face Hub - it's specifically designed for the Hugging Face Filesystem API. Also avoid it if you're working with Hugging Face Hub versions older than 0.14, as the Filesystem API dependency won't be satisfied.

You provide a path string pointing to a dataset file on Hugging Face Hub (for example, "datasets/dair-ai/emotion/data/data.jsonl.gz"). The loader accepts compressed formats like .jsonl.gz.

You receive one of three output types depending on the method called: LlamaIndex document objects from load_data, Python dictionaries from load_dicts, or pandas DataFrames from load_df. This flexibility lets you choose the data structure that matches your processing pipeline.

This loader integrates with LlamaIndex as a data reader component, designed specifically to feed data into LlamaIndex's document processing and indexing system. It relies on Hugging Face Hub's Filesystem API (version 0.14 or higher) to access datasets stored on the Hugging Face platform.

This tool is for developers building LlamaIndex applications who need to incorporate datasets from Hugging Face Hub. It's particularly useful for machine learning engineers and data scientists who want to leverage publicly available datasets in their RAG systems, search applications, or document processing pipelines without manual download steps.

Install the loader via pip:

pip install llama-index-readers-huggingface-fs

Basic usage example showing all three loading methods:

from pathlib import Path

from llama_index.readers.huggingface_fs import HuggingFaceFSReader

### load documents
loader = HuggingFaceFSReader()
documents = loader.load_data("datasets/dair-ai/emotion/data/data.jsonl.gz")

### load dicts
dicts = loader.load_dicts("datasets/dair-ai/emotion/data/data.jsonl.gz")

### load df
df = loader.load_df("datasets/dair-ai/emotion/data/data.jsonl.gz")