Load documents from Opensearch into LlamaIndex
Load documents from an OpenSearch index into LlamaIndex with an optional query DSL filter.
Why it matters
Retrieve and load documents from Opensearch indices into LlamaIndex for downstream processing, enabling developers to build RAG applications and AI workflows on top of their existing Opensearch data stores.
Outcomes
What it gets done
Connect to Opensearch clusters with authentication credentials
Execute JSON query DSL to filter and retrieve specific documents
Extract text content from designated fields in search results
Load documents with embeddings into LlamaIndex data structures
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-opensearch | bash Overview
Opensearch Loader
A LlamaIndex reader that loads a text field from OpenSearch-indexed documents, optionally filtered by a query DSL object. Use when you need existing OpenSearch-indexed content loaded into LlamaIndex, not for writing to the index.
What it does
The Opensearch Loader returns a set of texts corresponding to documents retrieved from an OpenSearch index. The user initializes the loader pointed at an OpenSearch index, then passes in a field to pull text from and, optionally, a JSON query DSL object to scope which documents get fetched.
OpensearchReader is initialized with the host, port, target index, and basic auth credentials. load_data then takes the field name to extract as document text, an optional query using OpenSearch's query DSL syntax, and an embedding_field naming which field holds vector embeddings if present.
When to use - and when NOT to
Use it when you need to pull a specific field's content from documents already indexed in OpenSearch into LlamaIndex, optionally filtered by a query DSL object rather than loading the entire index unfiltered. Do not use it as a way to write to or manage an OpenSearch index - it is a read-only loader for bringing existing indexed content into a LlamaIndex pipeline.
Capabilities
load_data fetches documents from the configured OpenSearch index, extracting a named text field and optionally an embedding field, filtered by an OpenSearch query DSL object.
How to install
pip install llama-index-readers-opensearch
Requires host, port, index name, and basic auth credentials for the target OpenSearch cluster.
Who it's for
Developers who already have data indexed in OpenSearch and want to load a specific, query-filtered set of it into LlamaIndex without re-indexing from the original source.
Source README
Opensearch Loader
pip install llama-index-readers-opensearch
The Opensearch Loader returns a set of texts corresponding to documents retrieved from an Opensearch index.
The user initializes the loader with an Opensearch index. They then pass in a field, and optionally a JSON query DSL object to fetch the fields they want.
Usage
Here's an example usage of the OpensearchReader to load 100 documents.
from llama_index.readers.opensearch import OpensearchReader
reader = OpensearchReader(
host="localhost",
port=9200,
index="<index_name>",
basic_auth=("<user_name>", "<password>"),
)
query = {"size": 100, "query": {"match_all": {}}}
documents = reader.load_data(
"<field_name>", query=query, embedding_field="field_name"
)
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.