Load Data from Cloud Storage
Load files from S3, Azure Blob, GCS, or any Apache OpenDAL-supported storage into LlamaIndex.
Why it matters
Effortlessly load data from various cloud storage services like S3, Azure Blob Storage, and Google Cloud Storage into your LlamaIndex applications. This asset simplifies data ingestion from distributed storage, making it readily available for AI processing.
Outcomes
What it gets done
Connect to and read data from S3 buckets.
Ingest files from Azure Blob Storage containers.
Retrieve data from Google Cloud Storage buckets.
Prepare cloud-stored data for use with LlamaIndex.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-opendal | bash Overview
OpenDAL Loaders
A family of LlamaIndex loaders for Apache OpenDAL-supported storage backends, with a general-purpose base reader and S3/Azblob/Gcs convenience readers. Use the dedicated provider reader for S3, Azure Blob, or GCS; use the base reader for any other OpenDAL-supported scheme.
What it does
The OpenDAL Loaders parse files from any storage service supported by Apache OpenDAL - including s3, azblob, gcs, and others - into LlamaIndex documents. Every loader in this family works the same way under the hood: files are temporarily downloaded locally, then parsed with SimpleDirectoryReader, so a custom file_extractor can be supplied to use any loader in the LlamaIndex ecosystem (or a custom one) for the actual file parsing.
The base OpendalReader takes a scheme (such as s3), a bucket, and a path, and load_data() downloads and parses everything under that path. Three convenience readers wrap the base reader for specific providers. OpendalAzblobReader is initialized with a container, path, endpoint, account_name, and account_key. OpendalGcsReader takes a bucket, path, endpoint, and credentials - if credentials is omitted, it falls back to loading from the environment. OpendalS3Reader (referred to as S3Reader in the source's credentials note) accepts a bucket, path, access_key_id, and secret_access_key, plus optional endpoint and region arguments to specify the S3 service's endpoint and region directly; if the access key ID or secret is omitted, it falls back to ~/.aws/credentials or environment variables, matching standard AWS credential resolution behavior. The source points to AWS's own IAM access-key documentation for how to obtain that access key and secret in the first place, rather than documenting AWS credential creation itself.
When to use - and when NOT to
Use the base OpendalReader when you need a storage backend OpenDAL supports but this package doesn't have a dedicated convenience reader for - just specify the right scheme. Use the dedicated OpendalAzblobReader, OpendalGcsReader, or OpendalS3Reader when working specifically with Azure Blob, Google Cloud Storage, or S3, since they pre-configure the scheme and expose provider-specific credential parameters directly. Do not supply explicit credentials in code for GCS or S3 if you'd rather rely on environment-based or ~/.aws/credentials-based resolution - both readers are designed to fall back automatically when credentials are omitted.
Capabilities
OpendalReader (scheme/bucket/path) is the general-purpose base loader. OpendalAzblobReader, OpendalGcsReader, and OpendalS3Reader are provider-specific convenience wrappers with their own credential parameters, all downloading files locally before parsing them with SimpleDirectoryReader (optionally with a custom file_extractor).
How to install
pip install llama-index-readers-opendal
Who it's for
Developers who need to load files from cloud object storage - S3, Azure Blob, GCS, or any other OpenDAL-supported backend - into LlamaIndex, using whichever file-format parser fits the downloaded content.
Source README
OpenDAL Loaders
pip install llama-index-readers-opendal
Base OpendalReader
This loader parses any file via Apache OpenDAL.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
Usage
OpendalReader can read data from any supported storage services including s3, azblob, gcs and so on.
from llama_index.readers.opendal import OpendalReader
loader = OpendalReader(
scheme="s3",
bucket="bucket",
path="path/to/data/",
)
documents = loader.load_data()
We also provide Opendal[S3|Gcs|Azblob]Reader for convenience.
This loader is designed to be used as a way to load data into LlamaIndex.
Azblob Loader
This loader parses any file stored on Azblob.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
Azblob loader is based on
OpendalReader.
Usage
from llama_index.readers.opendal import OpendalAzblobReader
loader = OpendalAzblobReader(
container="container",
path="path/to/data/",
endpoint="[endpoint]",
account_name="[account_name]",
account_key="[account_key]",
)
documents = loader.load_data()
This loader is designed to be used as a way to load data into LlamaIndex.
Gcs Loader
This loader parses any file stored on Gcs.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
Gcs loader is based on
OpendalReader.
Usage
from llama_index.readers.opendal import OpendalGcsReader
loader = OpendalGcsReader(
bucket="bucket",
path="path/to/data/",
endpoint="[endpoint]",
credentials="[credentials]",
)
documents = loader.load_data()
Note: if credentials is not provided, this loader to try to load from env.
This loader is designed to be used as a way to load data into LlamaIndex.
S3 Loader
This loader parses any file stored on S3. When initializing S3Reader, you may pass in your AWS Access Key. If none are found, the loader assumes they are stored in ~/.aws/credentials.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
S3 loader is based on
OpendalReader.
Usage
loader = OpendalS3Reader(
bucket="bucket",
path="path/to/data/",
access_key_id="[ACCESS_KEY_ID]",
secret_access_key="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
Note: if access_key_id or secret_access_key is not provided, this loader to try to load from env.
Possible arguments includes:
endpoint: Specify the endpoint of s3 service.region: Specify the region of s3 service.
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.