Load Elasticsearch Data into LlamaIndex

The Elasticsearch Reader integration for LlamaIndex retrieves documents from Elasticsearch or Opensearch indexes using the basic search API over REST. It loads data into LlamaIndex data structures and can be used as a Tool in a LangChain Agent.

Use this reader when you need to pull existing documents from an Elasticsearch or Opensearch index into a LlamaIndex workflow, especially when you want to leverage Elasticsearch's query DSL to filter and retrieve specific subsets of data.

Do NOT use this reader if your data source is not Elasticsearch or Opensearch.

You provide an Elasticsearch or Opensearch endpoint URL, an index name, a field name to extract text from, and an Elasticsearch JSON query DSL object to filter documents. Optionally, you can specify an embedding field if your documents already contain vector embeddings, and pass additional httpx client arguments like timeout settings.

You receive documents that can be utilized in downstream LlamaIndex data structures.

This reader is designed to load data into LlamaIndex and can subsequently be used as a Tool in a LangChain Agent.

Install via pip:

pip install llama-index-readers-elasticsearch

Basic usage example:

from llama_index.core.schema import Document
from llama_index.readers.elasticsearch import ElasticsearchReader

# Initialize ElasticsearchReader
reader = ElasticsearchReader(
    endpoint="<Your Elasticsearch/Opensearch Endpoint>",
    index="<Index Name>",
    httpx_client_args={
        "timeout": 10
    },  # Optional additional arguments for the httpx.Client
)

# Load data from Elasticsearch
documents = reader.load_data(
    field="<Field Name>",  # Field in the document to retrieve text from
    query={"query": {"match_all": {}}},  # Elasticsearch JSON query DSL object
    embedding_field="<Embedding Field>",  # Field for embeddings (optional)
)

This reader is built for developers and data engineers who maintain document collections in Elasticsearch or Opensearch and want to incorporate that data into LlamaIndex-powered applications.