Parse Documents with Dashscope Reader
Parse doc, docx, and PDF files into LlamaIndex documents via Alibaba's DashScope API.
Why it matters
Integrate Dashscope's document parsing capabilities into your LlamaIndex applications. Effortlessly extract and process information from various document types for advanced AI applications.
Outcomes
What it gets done
Load and parse documents (doc, docx, pdf) using the Dashscope API.
Configure parsing parameters like API keys, workspace IDs, and result types.
Process multiple files in parallel with configurable worker counts and timeouts.
Extract structured text data from documents for use in LlamaIndex pipelines.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-dashscope | bash Overview
LlamaIndex Readers Integration: Dashscope
A LlamaIndex reader that parses doc, docx, and PDF files into documents through Alibaba Cloud's DashScope parsing API. Use when you need cloud-based parsing of Word or PDF files rather than a local parsing library.
What it does
The Dashscope Reader parses document files - doc, docx, and PDF - into LlamaIndex Document objects using Alibaba Cloud's DashScope parsing API. DashScopeParse is configured with a result_type (defaulting to ResultType.DASHCOPE_DOCMIND) and given a list of file paths to parse; load_data returns a list of Document objects, each carrying the parsed text under text.
Parsing runs through DashScope's cloud API rather than locally, so the reader is a thin client: it manages authentication, sends the files for parsing, and waits for the result. Several settings control that process directly - num_workers (default 4, capped under 10) controls how many parsing requests run concurrently, check_interval (default 5 seconds) controls how often the reader polls for completion, and max_timeout (default 3600 seconds) bounds how long it will wait before giving up.
When to use - and when NOT to
Use it when you need to extract structured, indexable text from Word or PDF documents using a cloud parsing service rather than a local parsing library - particularly when documents are complex enough that a robust cloud parser is preferable to a lightweight local one. Do not use it for file types outside doc, docx, and PDF, and note it requires network access to DashScope's API and a valid API key or workspace ID rather than working fully offline.
Capabilities
load_data parses a file path or list of file paths (doc, docx, pdf) via the DashScope API and returns LlamaIndex Document objects. Configuration covers api_key, workspace_id, base_url (default https://dashscope.aliyuncs.com), result_type, worker concurrency, polling interval, timeout, verbosity, progress display, and whether to ignore per-file errors during parsing.
How to install
pip install llama-index-readers-dashscope
Requires a DashScope API key and workspace ID, either passed as parameters or set via environment variables.
Who it's for
Developers who need reliable extraction of text from Word or PDF documents into LlamaIndex, and prefer a managed cloud parsing service over a local parsing library.
Source README
LlamaIndex Readers Integration: Dashscope
Installation
pip install llama-index-readers-dashscope
Usage
from llama_index.readers.dashscope.base import DashScopeParse
from llama_index.readers.dashscope.utils import ResultType
file_list = [
# your files (accept doc, docx, pdf)
]
parse = DashScopeParse(result_type=ResultType.DASHCOPE_DOCMIND)
documents = parse.load_data(file_path=file_list)
Reader Setting:
A full list of retriever settings/kwargs is below:
- api_key: Optional[str] -- Your dashscope API key, which can be passed in through environment variables or parameters.
The parameter settings will override the results from the environment variables - workspace_id: Optional[str] -- Your dashscope workspace_id, which can be passed in through environment variables or
parameters. The parameter settings will override the results from the environment variables - base_url: Optional[str] -- The base url for the Dashscope API. The default value is "https://dashscope.aliyuncs.com".
The parameter settings will override the results from the environment variables. - result_type: Optional[ResultType] -- The result type for the parser. The default value is ResultType.DASHCOPE_DOCMIND.
- num_workers: Optional[int] -- The number of workers to use sending API requests for parsing. The default value is 4,
greater than 0, less than 10. - check_interval: Optional[int] -- The interval in seconds to check if the parsing is done. The default value is 5.
- max_timeout: Optional[int] -- The maximum timeout in seconds to wait for the parsing to finish. The default value is 3600.
- verbose: Optional[bool] -- Whether to print the progress of the parsing. The default value is True.
- show_progress: Optional[bool] -- Show progress when parsing multiple files. The default value is True.
- ignore_errors: Optional[bool] -- Whether or not to ignore and skip errors raised during parsing. The default value is
True.
Reader Input:
- file_path: Union[str, List[str]] -- The file path or list of file paths to parse.
Reader Output:
- List[llama_index.core.schema.Document] -- The list of documents parsed from the file.
- text: str -- The text of the document from DASHCOPE_DOCMIND.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.