Fetch and Process Document360 Articles
Fetch Document360 articles recursively into LlamaIndex with callback hooks and rate-limit handling.
Why it matters
Integrate your Document360 knowledge base into your AI applications. This reader fetches articles, handles complex API interactions, and prepares your documentation for retrieval-augmented generation.
Outcomes
What it gets done
Connect to the Document360 API using your credentials.
Recursively fetch articles from specified project versions and categories.
Process articles with customizable callbacks for error handling and data transformation.
Prepare fetched articles for use in RAG pipelines.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-document360 | bash Overview
LlamaIndex Readers Integration: Document360
A LlamaIndex reader that recursively fetches Document360 articles with extensive callback hooks and typed rate-limit/error handling. Use when a Document360 knowledge base needs loading with fine control over filtering, lifecycle events, and error handling.
What it does
The Document360Reader interacts with the Document360 API to fetch articles, processing them recursively while handling rate limiting and errors, and allows further customization through callback functions at nearly every stage of that process. Document360Reader requires an api_key, and calling load_data() walks the Document360 project structure, retrieving articles.
A long list of optional callbacks lets a developer hook into the process: should_process_project_version, should_process_category, and should_process_article each let the caller decide whether to process a given project version, category, or article; handle_category_processing_started and handle_article_processing_started fire when processing begins on each; handle_batch_finished fires once all articles are processed; handle_rate_limit_error, handle_request_http_error, and handle_load_data_error handle their respective error conditions; and article_to_custom_document lets a caller supply a custom transform from a Document360 article to a LlamaIndex document. Retry behavior for rate limits is configurable via rate_limit_num_retries (default 10) and rate_limit_retry_wait_time (default 30 seconds).
The reader also exposes typed entities (Article, ArticleSlim, Category, ProjectVersion) for use in the callback functions, and typed errors (RetryError, HTTPError, RateLimitException) that load_data() can raise and that calling code can catch specifically. The source notes that all fields on the entity types are marked Optional, because Document360's actual API responses sometimes don't match the schema documented in its own API docs.
When to use - and when NOT to
Use it when you need to pull a Document360 knowledge base into LlamaIndex, especially when you need fine-grained control over which project versions, categories, or articles get processed, or need to react to specific error types (rate limits versus HTTP errors versus other load failures) differently. Do not assume every field on a returned entity will be populated; treat all entity fields as optional given Document360's own documented API inconsistency.
Capabilities
load_data recursively fetches Document360 articles with configurable callbacks for filtering what gets processed, reacting to processing lifecycle events, and handling rate-limit or HTTP errors, plus a custom article-to-document transform and typed error classes for targeted exception handling.
How to install
pip install llama-index-readers-document360
Requires a Document360 API key.
Who it's for
Developers who need a Document360 knowledge base loaded into LlamaIndex with fine control over what gets processed and how errors and rate limits are handled.
Source README
LlamaIndex Readers Integration: Document360
The Document360Reader class is a custom reader that interacts with the Document360 API to fetch articles. It processes these articles recursively and allows further handling via custom callback functions, while also handling rate limiting and errors.
Installation
pip install llama-index-readers-document360
Usage
from document360_reader import Document360Reader
reader = Document360Reader(api_key="your_api_key")
### Load data
documents = reader.load_data()
### Use the documents as needed
for doc in documents:
print(doc.text)
Class Initialization
def __init__(
self,
api_key: str,
should_process_project_version=None,
should_process_category=None,
should_process_article=None,
handle_batch_finished=None,
handle_rate_limit_error=None,
handle_request_http_error=None,
handle_category_processing_started=None,
handle_article_processing_started=None,
handle_article_processing_error=None,
handle_load_data_error=None,
article_to_custom_document=None,
rate_limit_num_retries=10,
rate_limit_retry_wait_time=30,
):
pass
api_key: Your Document360 API key (required).should_process_project_version: Callback to determine whether to process a project version.should_process_category: Callback to determine whether to process a category.should_process_article: Callback to determine whether to process an article.handle_batch_finished: Callback executed after all articles are processed.handle_rate_limit_error: Callback for handling rate limit errors.handle_request_http_error: Callback for handling HTTP errors.handle_category_processing_started: Callback triggered when category processing starts.handle_article_processing_started: Callback triggered when article processing starts.handle_article_processing_error: Callback for handling errors during article processing.handle_load_data_error: Callback for handling errors during data loading.article_to_custom_document: Custom transformation function to map an article to a document.rate_limit_num_retries: Number of retry attempts when hitting rate limits.rate_limit_retry_wait_time: Time to wait (in seconds) between retries after a rate limit error.
Referencing entities
from llama_index.readers.document360.entities import (
Article,
ArticleSlim,
Category,
ProjectVersion,
)
def handle_category_processing_started(category: Category):
logging.info(f"Started processing category: {category}")
def handle_article_processing_started(article: Article):
logging.info(f"Processing article: {article}")
All the fields in the entities are marked as Optional. This is because the actual API responses from Document360 sometimes do not match the expected schema mentioned in the API documentation.
Referencing errors
from llama_index.readers.document360.errors import (
RetryError,
HTTPError,
RateLimitException,
)
reader = Document360Reader(api_key="your_api_key")
try:
reader.load_data()
except RetryError as e:
logging.info(f"Retry Error: {e}")
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.