Load S3 Files and Directories into LlamaIndex
S3Reader loads files or entire buckets from Amazon S3 into LlamaIndex, parsing content with SimpleDirectoryReader and optional custom file extractors.
Why it matters
Effortlessly ingest data from Amazon S3 into your LlamaIndex applications. This asset can load individual files or entire buckets with optional prefix filtering, enabling powerful RAG and data analysis pipelines.
Outcomes
What it gets done
Load specific files from S3 buckets.
Load all files within an S3 bucket or a specific prefix.
Integrate S3 data seamlessly with LlamaIndex for RAG.
Configure AWS credentials directly or via local configuration.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-s3 | bash Overview
S3 File or Directory Loader
S3 File or Directory Loader parses files stored in Amazon S3 buckets and loads them into LlamaIndex. It can process individual files by key, entire buckets, or files matching an optional prefix filter. All parsing is handled through SimpleDirectoryReader with support for custom file extractors. Use this loader when you need to load data from S3 into LlamaIndex. It supports prefix filtering for selective processing of subdirectories or file patterns.
What it does
S3 File or Directory Loader (S3Reader) parses files stored in Amazon S3 buckets and loads them into LlamaIndex for indexing and retrieval. It can process a single file by key, an entire bucket, or files matching an optional prefix filter. All files are parsed through SimpleDirectoryReader, allowing you to specify custom file extractors for different file types.
When to use - and when NOT to
Use S3Reader when you need to load data from S3 into LlamaIndex. Use the prefix filter when you only need to process specific subdirectories or file patterns within a bucket.
Do not use this loader if your files are stored locally or in other cloud storage providers - use the appropriate loader for those sources instead.
Inputs and outputs
You provide the S3 bucket name as a required input. Optionally, you can specify a file key for single-file loading, a prefix to filter which files to load, and AWS credentials (access key ID and secret). If credentials aren't provided during initialization, the loader looks for them in ~/.aws/credentials.
The loader returns documents via the load_data() method.
Integrations
This loader is designed to be used as a way to load data into LlamaIndex. It uses SimpleDirectoryReader under the hood, which means it's compatible with any custom file extractor available in the LlamaIndex readers library. AWS IAM credentials are supported for authentication.
Who it's for
S3Reader is built for developers and data engineers building LlamaIndex applications who store their document collections in Amazon S3.
pip install llama-index-readers-s3
loader = S3Reader(
bucket="scrabble-dictionary",
key="dictionary.txt",
aws_access_id="[ACCESS_KEY_ID]",
aws_access_secret="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
The loader handles nested file structures automatically - simply include the subdirectory path in the key parameter (e.g., subdirectory/input.txt). When no specific file key is provided, it processes the entire bucket or all files matching your prefix filter.
Source README
S3 File or Directory Loader
This loader parses any file stored on S3, or the entire Bucket (with an optional prefix filter) if no particular file is specified. When initializing S3Reader, you may pass in your AWS Access Key. If none are found, the loader assumes they are stored in ~/.aws/credentials.
All files are parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!
Installation
pip install llama-index-readers-s3
Usage
To use this loader, you need to pass in the name of your S3 Bucket. After that, if you want to just parse a single file, pass in its key. Note that if the file is nested in a subdirectory, the key should contain that, so like subdirectory/input.txt.
Otherwise, you may specify a prefix if you only want to parse certain files in the Bucket, or a subdirectory. AWS Access Key credentials may either be passed in during initialization or stored locally (see above).
loader = S3Reader(
bucket="scrabble-dictionary",
key="dictionary.txt",
aws_access_id="[ACCESS_KEY_ID]",
aws_access_secret="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.