Fetch and Process Document360 Articles
LlamaIndex reader that fetches articles from Document360 via API, with recursive processing, custom callbacks, and built-in rate limiting.
Why it matters
Integrate your Document360 knowledge base into your AI applications. This reader fetches articles, handles complex API interactions, and prepares your documentation for retrieval-augmented generation.
Outcomes
What it gets done
Connect to the Document360 API using your credentials.
Recursively fetch articles from specified project versions and categories.
Process articles with customizable callbacks for error handling and data transformation.
Prepare fetched articles for use in RAG pipelines.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-document360 | bash Overview
LlamaIndex Readers Integration: Document360
A LlamaIndex reader class for fetching and processing articles from Document360 via API Use for batch loading Document360 articles with custom processing control; avoid for real-time sync or when API responses deviate significantly from expected schema
What it does
The Document360Reader is a custom reader class that interacts with the Document360 API to fetch articles. It processes these articles recursively and allows further handling via custom callback functions, while also handling rate limiting and errors.
When to use - and when NOT to
Use this reader when you need to ingest articles from Document360 into your data pipeline. It provides callback hooks for custom handling of processing events, errors, and rate limits with configurable retry behavior. Do NOT use this if you need real-time synchronization - it's designed for batch loading. Note that entity fields are marked as optional because actual API responses from Document360 sometimes do not match the expected schema mentioned in the API documentation.
Inputs and outputs
You provide a Document360 API key (required) and optionally configure callback functions to control processing behavior. Optional callbacks include should_process_project_version, should_process_category, and should_process_article for filtering, plus handlers for batch completion, rate limits, HTTP errors, and custom document transformation via article_to_custom_document. Rate limit behavior is configurable through rate_limit_num_retries (default 10) and rate_limit_retry_wait_time (default 30 seconds).
The reader's load_data() method returns documents that you can iterate through:
from document360_reader import Document360Reader
reader = Document360Reader(api_key="your_api_key")
### Load data
documents = reader.load_data()
### Use the documents as needed
for doc in documents:
print(doc.text)
Integrations
The reader interacts with the Document360 API. You can reference entity types from the package:
from llama_index.readers.document360.entities import (
Article,
ArticleSlim,
Category,
ProjectVersion,
)
You can also reference error types for exception handling:
from llama_index.readers.document360.errors import (
RetryError,
HTTPError,
RateLimitException,
)
Who it's for
This integration serves developers building data pipelines on top of Document360. It's particularly valuable for teams who need programmatic control over processing through callbacks, or who need robust error handling with configurable retry logic for rate limits.
Installation:
pip install llama-index-readers-document360
Source README
LlamaIndex Readers Integration: Document360
The Document360Reader class is a custom reader that interacts with the Document360 API to fetch articles. It processes these articles recursively and allows further handling via custom callback functions, while also handling rate limiting and errors.
Installation
pip install llama-index-readers-document360
Usage
from document360_reader import Document360Reader
reader = Document360Reader(api_key="your_api_key")
### Load data
documents = reader.load_data()
### Use the documents as needed
for doc in documents:
print(doc.text)
Class Initialization
def __init__(
self,
api_key: str,
should_process_project_version=None,
should_process_category=None,
should_process_article=None,
handle_batch_finished=None,
handle_rate_limit_error=None,
handle_request_http_error=None,
handle_category_processing_started=None,
handle_article_processing_started=None,
handle_article_processing_error=None,
handle_load_data_error=None,
article_to_custom_document=None,
rate_limit_num_retries=10,
rate_limit_retry_wait_time=30,
):
pass
api_key: Your Document360 API key (required).should_process_project_version: Callback to determine whether to process a project version.should_process_category: Callback to determine whether to process a category.should_process_article: Callback to determine whether to process an article.handle_batch_finished: Callback executed after all articles are processed.handle_rate_limit_error: Callback for handling rate limit errors.handle_request_http_error: Callback for handling HTTP errors.handle_category_processing_started: Callback triggered when category processing starts.handle_article_processing_started: Callback triggered when article processing starts.handle_article_processing_error: Callback for handling errors during article processing.handle_load_data_error: Callback for handling errors during data loading.article_to_custom_document: Custom transformation function to map an article to a document.rate_limit_num_retries: Number of retry attempts when hitting rate limits.rate_limit_retry_wait_time: Time to wait (in seconds) between retries after a rate limit error.
Referencing entities
from llama_index.readers.document360.entities import (
Article,
ArticleSlim,
Category,
ProjectVersion,
)
def handle_category_processing_started(category: Category):
logging.info(f"Started processing category: {category}")
def handle_article_processing_started(article: Article):
logging.info(f"Processing article: {article}")
All the fields in the entities are marked as Optional. This is because the actual API responses from Document360 sometimes do not match the expected schema mentioned in the API documentation.
Referencing errors
from llama_index.readers.document360.errors import (
RetryError,
HTTPError,
RateLimitException,
)
reader = Document360Reader(api_key="your_api_key")
try:
reader.load_data()
except RetryError as e:
logging.info(f"Retry Error: {e}")
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.