Load data from Apache Iceberg tables into LlamaIndex
LlamaIndex reader that fetches data from Apache Iceberg tables into LlamaIndex workflows, supporting AWS profiles, regions, namespaces, and metadata column
Why it matters
Fetch and load data from Apache Iceberg tables into LlamaIndex for indexing and retrieval workflows, enabling users to build RAG applications on top of their data lake tables.
Outcomes
What it gets done
Connect to Apache Iceberg tables using AWS profile and region credentials
Specify namespace and table names to target specific datasets
Select metadata columns to include in the loaded documents
Transform Iceberg table data into LlamaIndex document format
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-iceberg | bash Overview
LlamaIndex Readers Integration: Iceberg
The Iceberg reader integration fetches data from Apache Iceberg tables and loads it into LlamaIndex. It connects using AWS profiles and regions, allowing you to specify namespaces, table names, and metadata columns to extract. Use this reader when you need to ingest data from Apache Iceberg tables into LlamaIndex.
What it does
The LlamaIndex Readers Integration for Iceberg loads data from Apache Iceberg tables directly into LlamaIndex workflows. It connects to Iceberg tables using AWS profiles and regional configurations, allowing you to specify namespaces, table names, and metadata columns to extract structured data for indexing and retrieval.
When to use - and when NOT to
Use this reader when you need to ingest data from Apache Iceberg tables into LlamaIndex. Do NOT use this reader if your data is not stored in Apache Iceberg format.
Inputs and outputs
You provide an AWS profile name, AWS region, Iceberg namespace, table name, and a list of metadata columns you want to include. The reader returns documents populated with the data from the specified Iceberg table.
How to use
Install the integration:
pip install llama-index-readers-iceberg
Load data from an Iceberg table:
from llama_index.readers.iceberg import IcebergReader
docs = IcebergReader().load_data(
profile_name="my_profile",
region="us-west-2",
namespace="my_dataset",
table="my_table",
metadata_columns=["_id", "_age", "_name"],
)
Who it's for
This reader is designed to be used as a way to load data into LlamaIndex.
Source README
LlamaIndex Readers Integration: Iceberg
pip install llama-index-readers-iceberg
This loader fetches data from Apache Iceberg tables.
Usage
To use this loader, you need to pass in an Intercom account access token.
from llama_index.readers.iceberg import IcebergReader
docs = IcebergReader().load_data(
profile_name="my_profile",
region="us-west-2",
namespace="my_dataset",
table="my_table",
metadata_columns=["_id", "_age", "_name"],
)
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.