Load Data from Cloud Storage
Load files from S3, Azure Blob, GCS, or any Apache OpenDAL-supported storage service into LlamaIndex.
Why it matters
Effortlessly load data from various cloud storage services like S3, Azure Blob Storage, and Google Cloud Storage into your LlamaIndex applications. This asset simplifies data ingestion from distributed storage, making it readily available for AI processing.
Outcomes
What it gets done
Connect to and read data from S3 buckets.
Ingest files from Azure Blob Storage containers.
Retrieve data from Google Cloud Storage buckets.
Prepare cloud-stored data for use with LlamaIndex.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-opendal | bash Overview
OpenDAL Loaders
A LlamaIndex reader family built on Apache OpenDAL, providing one consistent interface plus dedicated S3, Azure Blob Storage, and GCS readers for pulling files into LlamaIndex. Use it when source files live in S3, Azure Blob Storage, GCS, or another OpenDAL-supported service. Each provider reader has its own credential requirements, falling back to standard env-based auth when omitted.
What it does
llama-index-readers-opendal is a LlamaIndex reader package that parses files from any storage service supported by Apache OpenDAL, including S3, Azure Blob Storage (azblob), and Google Cloud Storage (gcs). The base OpendalReader takes a scheme, bucket, and path to read from any of these backends generically, and for convenience the package also ships dedicated OpendalS3Reader, OpendalAzblobReader, and OpendalGcsReader classes built on top of it.
When to use - and when NOT to
Use this reader when your source files live in S3, Azure Blob Storage, or GCS (or another OpenDAL-supported service) and you want one consistent reader interface across them rather than a separate library per cloud. Each provider-specific reader has its own credential requirements - S3 needs an AWS access key (falling back to ~/.aws/credentials if omitted), Azblob needs an account_name and account_key plus endpoint, and Gcs needs credentials or falls back to environment-based auth if none are supplied.
Inputs and outputs
All variants download files locally and parse them with SimpleDirectoryReader under the hood, so you can pass a custom file_extractor relying on any loader in the library or your own:
loader = OpendalReader(
scheme="s3",
bucket="bucket",
path="path/to/data/",
)
documents = loader.load_data()
The S3-specific reader also accepts optional endpoint and region arguments to target a non-default S3-compatible service or region:
loader = OpendalS3Reader(
bucket="bucket",
path="path/to/data/",
access_key_id="[ACCESS_KEY_ID]",
secret_access_key="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
Integrations
The Azblob and Gcs readers follow the same load_data() pattern, differing only in their constructor arguments: OpendalAzblobReader takes container (Azure's term for its top-level bucket-equivalent), path, endpoint, account_name, and account_key; OpendalGcsReader takes bucket, path, endpoint, and optional credentials. All three provider readers are built on the same OpendalReader base, so switching storage backends means swapping the reader class and its credentials, not rewriting your ingestion logic. Every variant is designed to be used as a way to load data into LlamaIndex, not as a standalone file browser. Install with pip install llama-index-readers-opendal.
Who it's for
Developers building LlamaIndex pipelines that need to read from S3, Azure Blob Storage, or GCS with one shared reader pattern, rather than integrating a separate SDK per cloud provider. Because every provider reader is a thin wrapper over the same OpendalReader base and the same SimpleDirectoryReader parsing step underneath, adding support for a new file type or a new OpenDAL-backed storage service doesn't require changing how downstream documents are consumed - only how they're fetched.
Source README
OpenDAL Loaders
pip install llama-index-readers-opendal
Base OpendalReader
This loader parses any file via Apache OpenDAL.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
Usage
OpendalReader can read data from any supported storage services including s3, azblob, gcs and so on.
from llama_index.readers.opendal import OpendalReader
loader = OpendalReader(
scheme="s3",
bucket="bucket",
path="path/to/data/",
)
documents = loader.load_data()
We also provide Opendal[S3|Gcs|Azblob]Reader for convenience.
This loader is designed to be used as a way to load data into LlamaIndex.
Azblob Loader
This loader parses any file stored on Azblob.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
Azblob loader is based on
OpendalReader.
Usage
from llama_index.readers.opendal import OpendalAzblobReader
loader = OpendalAzblobReader(
container="container",
path="path/to/data/",
endpoint="[endpoint]",
account_name="[account_name]",
account_key="[account_key]",
)
documents = loader.load_data()
This loader is designed to be used as a way to load data into LlamaIndex.
Gcs Loader
This loader parses any file stored on Gcs.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
Gcs loader is based on
OpendalReader.
Usage
from llama_index.readers.opendal import OpendalGcsReader
loader = OpendalGcsReader(
bucket="bucket",
path="path/to/data/",
endpoint="[endpoint]",
credentials="[credentials]",
)
documents = loader.load_data()
Note: if credentials is not provided, this loader to try to load from env.
This loader is designed to be used as a way to load data into LlamaIndex.
S3 Loader
This loader parses any file stored on S3. When initializing S3Reader, you may pass in your AWS Access Key. If none are found, the loader assumes they are stored in ~/.aws/credentials.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
S3 loader is based on
OpendalReader.
Usage
loader = OpendalS3Reader(
bucket="bucket",
path="path/to/data/",
access_key_id="[ACCESS_KEY_ID]",
secret_access_key="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
Note: if access_key_id or secret_access_key is not provided, this loader to try to load from env.
Possible arguments includes:
endpoint: Specify the endpoint of s3 service.region: Specify the region of s3 service.
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.