Load Zulip Stream Data
Load messages from Zulip streams into LlamaIndex documents using a bot API token.
Why it matters
Ingest and structure conversational data from Zulip streams for use in AI applications. This asset connects to your Zulip instance to extract messages, making them available for analysis, indexing, or further processing.
Outcomes
What it gets done
Connect to Zulip using bot credentials.
Fetch messages from specified or all Zulip streams.
Format stream content into a list of documents.
Prepare data for LlamaIndex or LangChain agents.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-zulip | bash Overview
Initialize the ZulipReader with the bot's email and Zulip domain
A LlamaIndex reader that loads Zulip stream messages via a bot API token, scoped to specific streams or all streams. Use to search or ask questions over a Zulip workspace's message history, given a configured bot token.
What it does
The Zulip Loader loads data from Zulip streams into LlamaIndex documents using a Zulip bot's API token. It fetches messages from specified streams, or from all streams if none are named, and returns a list of documents containing the content of those streams.
Before use, a Zulip bot must be created and its API token obtained through Zulip's own bot-creation process, then set as the ZULIP_TOKEN environment variable. ZulipReader is initialized with the bot's email and the Zulip domain, and load_data is called either with get_all_streams() to pull every stream, or with an explicit list of stream names to scope the load to just those. A reverse_chronological parameter can also be passed to load_data to control whether messages come back oldest-first or newest-first.
When to use - and when NOT to
Use it when you want to build a LlamaIndex index over the message history of one or more Zulip streams - for example, to search or ask questions about what was discussed in a team's Zulip workspace. Do not use it without first setting up a Zulip bot and granting it access to the streams you want to read; the reader has no access beyond what the bot's API token permits.
Capabilities
load_data fetches messages from named Zulip streams (or all streams via get_all_streams()) and returns them as LlamaIndex documents, with an optional reverse_chronological ordering parameter.
How to install
pip install llama-index-readers-zulip
Requires a Zulip bot's email, its Zulip domain, and its API token set as the ZULIP_TOKEN environment variable.
Who it's for
Developers who want to search or ask questions over a Zulip workspace's message history, scoped to specific streams or the whole organization.
Source README
Zulip Loader
The Zulip Loader is a Python script that allows you to load data from Zulip streams using a Zulip bot's API token. It fetches messages from specified streams or all streams if none are specified, and returns a list of documents with the stream content.
Prerequisites
Create a Zulip bot and obtain its API token. Follow the instructions in the Zulip documentation to create a bot and get the API key (token).
Set the ZULIP_TOKEN environment variable to your Zulip bot's API token:
export ZULIP_TOKEN="your-zulip-bot-api-token"
Installation
You can install the Zulip Reader via pip:
pip install llama-index-readers-zulip
Usage
Use the ZulipReader class to load data from Zulip streams:
from zulip_loader import ZulipReader
### Initialize the ZulipReader with the bot's email and Zulip domain
reader = ZulipReader(
zulip_email="your-bot-email@your-zulip-domain.zulipchat.com",
zulip_domain="your-zulip-domain.zulipchat.com",
)
### Load data from all streams
data = reader.load_data(reader.get_all_streams())
### Load data from specific streams
stream_names = ["stream1", "stream2"]
data = reader.load_data(stream_names)
### This will return a list of documents containing the content of the specified streams.
For more customization, you can pass the reverse_chronological parameter to the load_data() method to indicate the order of messages in the output.
This loader is designed to be used as a way to load data into
LlamaIndex and/or subsequently
used as a Tool in a LangChain Agent.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.