Load Remote Data from URLs
LlamaIndex reader that extracts text from any remote page or file by URL.
Why it matters
Effortlessly ingest text content from any web page or file URL. This asset acts as a universal data loader, capable of parsing various file types and even transcribing audio from provided links.
Outcomes
What it gets done
Download and parse content from any given URL.
Support for various file types including text, images, and audio.
Integrates seamlessly with LlamaIndex for further data processing.
Temporarily stores downloaded files for parsing.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-remote | bash Overview
Remote Page/File Loader
The Remote Page/File Loader extracts text from any remote page or file by URL, downloading and parsing files behind the scenes via SimpleDirectoryReader, or AudioTranscriber for audio. Use it when you need text extracted from a URL without handling its file type yourself. A custom file_extractor can override default parsing for specific types.
What it does
The Remote Page/File Loader extracts text from any remote page or file, given just its URL. If a file lives at the URL, it downloads it temporarily and parses it with SimpleDirectoryReader, so it is effectively an all-in-one tool for almost any URL: a .txt URL (like a Project Gutenberg book) is parsed as-is, while a hosted .mp3 URL is downloaded and parsed with AudioTranscriber.
When to use - and when NOT to
Use it when you need to extract text from a URL without knowing or handling its file type yourself -- web pages, text files, audio files, or anything else SimpleDirectoryReader and its extractors can parse. You can optionally supply a custom file_extractor to override the default parsing for a given type.
Inputs and outputs
Install with:
pip install llama-index-readers-remote
Load data by URL:
from llama_index.readers.remote import RemoteReader
loader = RemoteReader()
documents = loader.load_data(
url="https://en.wikipedia.org/wiki/File:Example.jpg"
)
load_data takes a url and returns the parsed content as documents, downloading and parsing files behind the scenes when the URL points to one.
Who it's for
Developers building LlamaIndex pipelines that need to pull text out of arbitrary URLs -- pages, text files, audio files -- without writing per-file-type handling themselves.
Source README
Remote Page/File Loader
pip install llama-index-readers-remote
This loader makes it easy to extract the text from any remote page or file using just its url. If there's a file at the url, this loader will download it temporarily and parse it using SimpleDirectoryReader. It is an all-in-one tool for (almost) any url.
As a result, any page or type of file is supported. For instance, if a .txt url such as a Project Gutenberg book is passed in, the text will be parsed as is. On the other hand, if a hosted .mp3 url is passed in, it will be downloaded and parsed using AudioTranscriber.
Usage
To use this loader, you need to pass in a Path to a local file. Optionally, you may specify a file_extractor for the SimpleDirectoryReader to use, other than the default one.
from llama_index.readers.remote import RemoteReader
loader = RemoteReader()
documents = loader.load_data(
url="https://en.wikipedia.org/wiki/File:Example.jpg"
)
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.