Load Web Data with Apify Actors
Apify Loaders for LlamaIndex load web-scraped data from Apify Actors and datasets into LlamaIndex documents for RAG pipelines and vector indexing.
Why it matters
Integrate with Apify's powerful web scraping platform to extract and load data from web pages or existing datasets directly into your LlamaIndex or LangChain applications.
Outcomes
What it gets done
Run pre-built Apify Actors to crawl and extract content from websites.
Load data from existing Apify datasets.
Prepare scraped data for use in LLM applications via LlamaIndex.
Utilize Apify data within LangChain agents.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-apify | bash Overview
Apify Loaders
Apify Loaders include the ApifyActor loader that runs a specific Actor and loads its results, and the ApifyDataset loader that loads documents from an existing Apify dataset. Use Apify Loaders when you want to run Apify Actors or load existing Apify datasets into LlamaIndex. The source material shows examples of crawling documentation sites and extracting text content that can be fed to a vector index or language model like GPT.
What it does
Apify Loaders includes the ApifyActor loader that runs a specific Apify Actor and loads its results, and the ApifyDataset loader that loads documents from an existing Apify dataset. The source material demonstrates using these loaders with the Website Content Crawler Actor, which can deeply crawl websites such as documentation, knowledge bases, help centers, or blogs, and extract text content from web pages. The extracted text can be fed to a vector index or language model like GPT in order to answer questions from it.
When to use - and when NOT to
Use Apify Loaders when you want to run Apify Actors or load existing Apify datasets into LlamaIndex. The source material shows examples of crawling documentation sites and extracting text content. Do NOT use these loaders if you already have structured data in a database or file system - standard LlamaIndex readers will be more direct.
Inputs and outputs
The ApifyActor loader requires an Apify API token, an actor_id parameter, run_input parameters, and a dataset_mapping_function. The ApifyDataset loader requires an Apify API token, a dataset_id parameter, and a dataset_mapping_function. The source material shows the dataset_mapping_function creating Document objects with text and metadata fields.
Integrations
This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. Apify is a cloud platform for web scraping and data extraction, which provides an ecosystem of more than a thousand ready-made apps called Actors for various scraping, crawling, and extraction use cases.
Installation and usage
Install via pip:
pip install llama-index-readers-apify
Example using ApifyActor to crawl and load website content:
from llama_index.core import Document
from llama_index.readers.apify import ApifyActor
reader = ApifyActor("<My Apify API token>")
documents = reader.load_data(
actor_id="apify/website-content-crawler",
run_input={
"startUrls": [{"url": "https://docs.llamaindex.ai/en/latest/"}]
},
dataset_mapping_function=lambda item: Document(
text=item.get("text"),
metadata={
"url": item.get("url"),
},
),
)
Who it's for
Apify Loaders are for developers who want to load data from Apify into LlamaIndex. To use this loader, you need to have a (free) Apify account and set your Apify API token in the code.
Source README
Apify Loaders
pip install llama-index-readers-apify
Apify Actor Loader
Apify is a cloud platform for web scraping and data extraction,
which provides an ecosystem of more than a thousand
ready-made apps called Actors for various scraping, crawling, and extraction use cases.
This loader runs a specific Actor and loads its results.
Usage
In this example, we’ll use the Website Content Crawler Actor,
which can deeply crawl websites such as documentation, knowledge bases, help centers,
or blogs, and extract text content from the web pages.
The extracted text then can be fed to a vector index or language model like GPT
in order to answer questions from it.
To use this loader, you need to have a (free) Apify account
and set your Apify API token in the code.
from llama_index.core import Document
from llama_index.readers.apify import ApifyActor
reader = ApifyActor("<My Apify API token>")
documents = reader.load_data(
actor_id="apify/website-content-crawler",
run_input={
"startUrls": [{"url": "https://docs.llamaindex.ai/en/latest/"}]
},
dataset_mapping_function=lambda item: Document(
text=item.get("text"),
metadata={
"url": item.get("url"),
},
),
)
This loader is designed to be used as a way to load data into
LlamaIndex and/or subsequently
used as a Tool in a LangChain Agent.
Apify Dataset Loader
Apify is a cloud platform for web scraping and data extraction,
which provides an ecosystem of more than a thousand
ready-made apps called Actors for various scraping, crawling, and extraction use cases.
This loader loads documents from an existing Apify dataset.
Usage
In this example, we’ll load a dataset generated by
the Website Content Crawler Actor,
which can deeply crawl websites such as documentation, knowledge bases, help centers,
or blogs, and extract text content from the web pages.
The extracted text then can be fed to a vector index or language model like GPT
in order to answer questions from it.
To use this loader, you need to have a (free) Apify account
and set your Apify API token in the code.
from llama_index.core import Document
from llama_index.readers.apify import ApifyDataset
reader = ApifyDataset("<Your Apify API token>")
documents = reader.load_data(
dataset_id="<Apify Dataset ID>",
dataset_mapping_function=lambda item: Document(
text=item.get("text"),
metadata={
"url": item.get("url"),
},
),
)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.