Extract and Load Webpage Content Recursively
Recursively extract text from a webpage's linked pages and files into LlamaIndex.
Why it matters
Automate the extraction of text and links from a given webpage URL, including content from linked pages up to a specified depth. This asset is ideal for researchers and content curators needing to gather and process information from complex web structures.
Outcomes
What it gets done
Recursively scrape linked pages up to a defined depth.
Extract text content from various file types found on webpages (e.g., PDFs).
Load extracted data into LlamaIndex for further processing.
Identify and extract all hyperlinks present on a webpage.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-remote-depth | bash Overview
Remote Page/File Loader
A LlamaIndex reader that recursively extracts text from a webpage's linked pages and files up to a configurable link depth. Use for link-heavy starting points like a syllabus or documentation index, not for a single standalone page.
What it does
The Remote Page/File Loader extracts text from the links available on a webpage URL, and can follow the links found within those linked pages to extract further links - functioning as an all-in-one tool for loading content from almost any group of related URLs. It is built on RemoteReader (which reads a single page) which is itself built on SimpleDirectoryReader (which parses the downloaded content if it's a file type like PDF).
RemoteDepthReader is called with load_data(url=...), and a depth parameter controls how many levels of links to follow - depth=2 means extracting the links on the starting page, then also extracting the links found within those linked pages. The source's own example points at an MIT OpenCourseWare syllabus page, noting it will pull the syllabus itself along with linked PDFs and other resources.
When to use - and when NOT to
Use it when you want to load not just a single webpage but everything linked from it - a course syllabus with its linked PDFs, a documentation index with its sub-pages, or similar link-heavy starting points - without manually enumerating each URL yourself. Set depth deliberately: a higher depth pulls in more content but also more links, which can grow quickly if the linked pages are themselves link-heavy. Do not use it for a single, standalone page with no useful outbound links; a plain page reader is a better fit there.
Capabilities
load_data starts at a given URL and recursively extracts text from linked pages and files (via SimpleDirectoryReader-style parsing) up to a configurable link depth.
How to install
pip install llama-index-readers-remote-depth
Who it's for
Developers who need to load content from a whole cluster of linked pages and files starting from one URL, rather than a single page in isolation.
Source README
Remote Page/File Loader
pip install llama-index-readers-remote-depth
This loader makes it easy to extract the text from the links available in a webpage URL, and extract the links presents in the page. It's based on RemoteReader (reading single page), that is based on SimpleDirectoryReader (parsing the document if file is a pdf, etc). It is an all-in-one tool for (almost) any group of urls.
You can try with this MIT lecture link, it will be able to extract the syllabus, the PDFs, etc:https://ocw.mit.edu/courses/5-05-principles-of-inorganic-chemistry-iii-spring-2005/pages/syllabus/
Usage
You need to specify the parameter depth to specify how many levels of links you want to extract. For example, if you want to extract the links in the page, and the links in the links in the page, you need to specify depth=2.
from llama_index.readers.remote_depth import RemoteDepthReader
loader = RemoteDepthReader()
documents = loader.load_data(
url="https://ocw.mit.edu/courses/5-05-principles-of-inorganic-chemistry-iii-spring-2005/pages/syllabus/"
)
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.