Read Structured Data Files for LlamaIndex
LlamaIndex reader that loads JSON, JSONL, CSV, and XLSX files into documents with configurable column-to-text and column-to-metadata mapping.
Why it matters
Integrate structured data files (JSON, CSV, XLSX) into your LlamaIndex applications. This asset enables efficient data loading and parsing for advanced AI use cases.
Outcomes
What it gets done
Load data from JSON, JSONL, CSV, and XLSX files.
Differentiate between main text content and metadata columns.
Integrate with LlamaIndex for document ingestion.
Process single files or entire directories of structured data.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-structured-data | bash Overview
LlamaIndex Readers Integration: Structured-Data
StructuredDataReader is a LlamaIndex integration that reads JSON, JSONL, CSV, and XLSX files and converts them into LlamaIndex documents. It uses col_index and col_metadata parameters to specify which columns become searchable text and which become metadata, giving you precise control over how tabular data is indexed. Use this reader when you need to ingest structured data files into a LlamaIndex pipeline with explicit control over column mapping. It works for both single files and batch directory processing, making it ideal for indexing product catalogs, datasets, or spreadsheet exports for semantic search.
What it does
StructuredDataReader is a LlamaIndex integration that ingests structured data files (JSON, JSONL, CSV, XLSX) and converts them into LlamaIndex documents. It provides col_index and col_metadata parameters to control which columns become document text and which become metadata, enabling precise control over how tabular data is transformed for indexing and retrieval.
When to use - and when NOT to
Use StructuredDataReader when you need to ingest tabular or structured data files into a LlamaIndex pipeline and want explicit control over which columns contribute to searchable text versus metadata fields. It supports processing directories of multiple structured file types with a single parser configuration.
Inputs and outputs
You provide file paths (as Path objects) to JSON, JSONL, CSV, or XLSX files, along with col_index parameters specifying which columns to include in document text (by name or integer index) and col_metadata parameters identifying columns to store as metadata. The reader outputs LlamaIndex Document objects with the specified columns mapped to text content and metadata fields. Column specifications accept lists of column names, integer indices (including negative indexing), or single values.
Integrations
StructuredDataReader integrates with LlamaIndex SimpleDirectoryReader for batch processing of directories containing multiple structured data files. You configure it as a file extractor mapped to specific extensions (.xlsx, .csv, .json, .jsonl), allowing SimpleDirectoryReader to automatically apply the parser to matching files.
Who it's for
This reader is for developers building LlamaIndex applications that need to ingest structured data files with column-level control over text versus metadata mapping.
Installation
pip install llama-index-readers-structured-data
Single file example
from pathlib import Path
from llama_index.readers.structured_data.base import StructuredDataReader
parser = StructuredDataReader(col_index=["col1", "col2"], col_metadata=0)
documents = parser.load_data(Path("your/file/path.json"))
Directory processing example
from pathlib import Path
from llama_index.core import SimpleDirectoryReader
from llama_index.readers.structured_data.base import StructuredDataReader
parser = StructuredDataReader(col_index=[1, -1], col_metadata="col3")
file_extractor = {
".xlsx": parser,
".csv": parser,
".json": parser,
".jsonl": parser,
}
documents = SimpleDirectoryReader(
"your/dic/path", file_extractor=file_extractor
).load_data()
Source README
LlamaIndex Readers Integration: Structured-Data
The function 'StructuredDataReader' supports reading files in JSON, JSONL, CSV, and XLSX formats. It provides parameters 'col_index' and 'col_metadata' to differentiate between columns that should be written into the document's main text and additional metadata.
Install package
pip install llama-index-readers-structured-data
Or install locally:
pip install -e llama-index-integrations/readers/llama-index-readers-structured-data
Usage
- for single document:
from pathlib import Path
from llama_index.readers.structured_data.base import StructuredDataReader
parser = StructuredDataReader(col_index=["col1", "col2"], col_metadata=0)
documents = parser.load_data(Path("your/file/path.json"))
- for dictory of documents:
from pathlib import Path
from llama_index.core import SimpleDirectoryReader
from llama_index.readers.structured_data.base import StructuredDataReader
parser = StructuredDataReader(col_index=[1, -1], col_metadata="col3")
file_extractor = {
".xlsx": parser,
".csv": parser,
".json": parser,
".jsonl": parser,
}
documents = SimpleDirectoryReader(
"your/dic/path", file_extractor=file_extractor
).load_data()
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.