Query Pathway vector store for semantic document retrieval
Retrieve nearest-neighbor documents from a Pathway vector store into LlamaIndex.
Why it matters
Retrieve semantically relevant documents from a Pathway data indexing pipeline by querying the vector store with natural language text and returning the closest matching neighbors.
Outcomes
What it gets done
Connect to Pathway server using host and port configuration
Query vector store with natural language text to find similar documents
Filter results by metadata criteria to narrow search scope
Return top-k nearest neighbor documents for downstream processing
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-pathway | bash Overview
LlamaIndex Readers Integration: Pathway
A LlamaIndex reader that queries a running Pathway server's vector store for the nearest-neighbor documents to a text query. Use when you already run a Pathway indexing pipeline and want its results as LlamaIndex documents.
What it does
The Pathway Reader is a utility class for retrieving documents from the Pathway data indexing pipeline. It queries the Pathway vector store to get the closest neighbors of a given text query, functioning as a bridge between an already-running Pathway indexing pipeline and LlamaIndex's document format.
PathwayReader is initialized with the host and port of a running Pathway server, and load_data takes a query_text (the text to find the closest neighbors of), a k parameter controlling how many results to return, and an optional metadata_filter to narrow the search further.
When to use - and when NOT to
Use it when you already run a Pathway data indexing pipeline and want its nearest-neighbor search results available as LlamaIndex documents - for example, feeding Pathway-indexed content into a broader LlamaIndex retrieval or query workflow. Do not use it without an already-running Pathway server; the reader is a thin client over Pathway's vector store, not a standalone indexing pipeline itself.
Capabilities
load_data queries a running Pathway server's vector store for the k closest neighbors of a text query, optionally filtered by metadata, and returns the results as LlamaIndex documents.
How to install
pip install llama-index-readers-pathway
Requires a reachable, already-running Pathway server (host and port).
Who it's for
Developers who already operate a Pathway indexing pipeline and want its nearest-neighbor search results integrated into a LlamaIndex retrieval or query workflow.
Source README
LlamaIndex Readers Integration: Pathway
Overview
Pathway Reader is a utility class for retrieving documents from the Pathway data indexing pipeline. It queries the Pathway vector store to get the closest neighbors of a given text query.
Installation
You can install Pathway Reader via pip:
pip install llama-index-readers-pathway
Usage
from llama_index.readers.pathway import PathwayReader
### Initialize PathwayReader with the URI and port of the Pathway server
reader = PathwayReader(host="<Pathway Host>", port="<Port>")
### Load data from Pathway
documents = reader.load_data(
query_text="<Query Text>", # The text to get the closest neighbors of
k=4, # Number of results to return
metadata_filter="<Metadata Filter>", # Filter to be applied
)
This loader is designed to be used as a way to load data into
LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.