Load Azure Blob Storage Data into LlamaIndex
LlamaIndex reader that loads files or containers from Azure Storage Blob.
Why it matters
Effortlessly ingest data from Azure Blob Storage into your LlamaIndex applications. This asset handles authentication and data extraction, preparing your information for AI-driven analysis and retrieval.
Outcomes
What it gets done
Connect to Azure Blob Storage using SAS tokens, connection strings, or Azure AD credentials.
Load individual files or entire containers with optional prefix filtering.
Temporarily download and parse files using LlamaIndex's SimpleDirectoryReader.
Support for custom file extractors to handle diverse data formats.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-azstorage-blob | bash Overview
Azure Storage Blob Loader
The Azure Storage Blob Loader parses files or entire containers from Azure Blob Storage into LlamaIndex documents, authenticated via a SAS URL, connection string, or Azure AD, with support for custom per-file-type extractors. Use it when you need Azure Storage blob content -- a single file, a filtered subset, or a whole container -- loaded into LlamaIndex. It requires one of three supported authentication methods.
What it does
The Azure Storage Blob Loader parses any file stored as an Azure Storage blob, or an entire container (optionally filtered by prefix or attribute) if no specific file is given. It is a thin wrapper over the Azure Blob Storage Client for Python, specifically its ContainerClient, so any parameter that client accepts is available here too. Files are temporarily downloaded locally and parsed with SimpleDirectoryReader, so you can also pass a custom file_extractor to route specific file types to any loader in the library, or your own.
When to use - and when NOT to
Use it when you need one file, a filtered subset, or an entire Azure Storage container loaded into LlamaIndex documents. It supports three authentication methods -- a SAS URL, a storage account connection string, or Azure AD (via DefaultAzureCredential or another credential like ClientSecretCredential) -- so it is not usable without one of those already configured for your Azure account.
Inputs and outputs
Install with:
pip install llama-index-readers-azstorage-blob
Using a SAS URL for a single blob:
from llama_index.readers.azstorage_blob import AzStorageBlobReader
loader = AzStorageBlobReader(
container="scrabble-dictionary",
blob="dictionary.txt",
account_url="<SAS_URL>",
)
documents = loader.load_data()
Using a connection string to pull an entire container:
from llama_index.readers.azstorage_blob import AzStorageBlobReader
loader = AzStorageBlobReader(
container_name="<CONTAINER_NAME>",
connection_string="<STORAGE_ACCOUNT_CONNECTION_STRING>",
)
documents = loader.load_data()
Using Azure AD (requires pip install azure-identity):
from azure.identity import DefaultAzureCredential
default_credential = DefaultAzureCredential()
from llama_index.readers.azstorage_blob import AzStorageBlobReader
loader = AzStorageBlobReader(
container_name="scrabble-dictionary",
account_url="https://<storage account name>.blob.core.windows.net",
credential=default_credential,
)
documents = loader.load_data()
A nested blob's name should include its path, e.g. subdirectory/input.txt. To use a custom file extractor, update the DEFAULT_FILE_READER_CLS dict, for example mapping .pdf to your own SimplePDFReader() instance. A 2023-12-14 update added the connection-string authentication method shown above and changed how temporarily-downloaded files are named, reverting from random names back to the files' original names.
Who it's for
Developers building LlamaIndex pipelines on Azure Storage content, who need flexibility in both authentication method (SAS URL, connection string, or Azure AD) and file parsing, via a custom file_extractor for formats beyond what SimpleDirectoryReader handles by default.
Source README
Azure Storage Blob Loader
pip install llama-index-readers-azstorage-blob
This loader parses any file stored as an Azure Storage blob or the entire container (with an optional prefix / attribute filter) if no particular file is specified. When initializing AzStorageBlobReader, you may pass in your account url with a SAS token or crdentials to authenticate.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)! If you need a clue on finding the file extractor object because you'd like to use your own file extractor, follow this sample.
import llama_index
file_extractor = llama_index.readers.file.base.DEFAULT_FILE_READER_CLS
### Make sure to use an instantiation of a class
file_extractor.update({".pdf": SimplePDFReader()})
Usage
To use this loader, you need to pass in the name of your Azure Storage Container. After that, if you want to just parse a single file, pass in its blob name. Note that if the file is nested in a subdirectory, the blob name should contain the path such as subdirectory/input.txt. This loader is a thin wrapper over the Azure Blob Storage Client for Python, see ContainerClient for detailed parameter usage options.
Using a Storage Account SAS URL
from llama_index.readers.azstorage_blob import AzStorageBlobReader
loader = AzStorageBlobReader(
container="scrabble-dictionary",
blob="dictionary.txt",
account_url="<SAS_URL>",
)
documents = loader.load_data()
Using a Storage Account with connection string
The sample below will download all files in a container, by only specifying the storage account's connection string and the container name.
from llama_index.readers.azstorage_blob import AzStorageBlobReader
loader = AzStorageBlobReader(
container_name="<CONTAINER_NAME>",
connection_string="<STORAGE_ACCOUNT_CONNECTION_STRING>",
)
documents = loader.load_data()
Using Azure AD
Ensure the Azure Identity library is available pip install azure-identity
The sample below downloads all files in the container using the default credential, alternative credential options are available such as a service principal ClientSecretCredential
from azure.identity import DefaultAzureCredential
default_credential = DefaultAzureCredential()
from llama_index.readers.azstorage_blob import AzStorageBlobReader
loader = AzStorageBlobReader(
container_name="scrabble-dictionary",
account_url="https://<storage account name>.blob.core.windows.net",
credential=default_credential,
)
documents = loader.load_data()
This loader is designed to be used as a way to load data into LlamaIndex.
Updates
[2023-12-14] by JAlexMcGraw (#765)
- Added functionality to allow user to connect to blob storage with connection string
- Changed temporary file names from random to back to original names
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.