Scrape Web Data with Oxylabs
Load Google Search results and other scraped web data into LlamaIndex via Oxylabs.
Why it matters
Integrate Oxylabs' powerful web scraping capabilities directly into your LlamaIndex applications. Effortlessly extract data from search engines, e-commerce sites, and any other website for your AI projects.
Outcomes
What it gets done
Connect to Oxylabs API using your credentials.
Configure search parameters like query and geo-location.
Load and process data scraped from various websites.
Utilize scraped data within your LlamaIndex pipelines.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-oxylabs | bash Overview
LlamaIndex Oxylabs Reader Integration
A LlamaIndex reader that loads Google Search results and other web data via the Oxylabs Web Scraper API. Use when you need scraped or geo-localized search results loaded into LlamaIndex without building your own scraper.
What it does
The Oxylabs Reader Integration loads data from search engines, e-commerce sites, travel platforms, and other websites into LlamaIndex documents using the Oxylabs Web Scraper API. OxylabsGoogleSearchReader is one of the readers in the package, initialized with Oxylabs account credentials, and its load_data method takes a query dictionary - for example a search query string, a parse flag to have Oxylabs parse the result into structured data, and a geo_location to run the search as if from a specific place.
The integration can be installed with pip, poetry, or uv, reflecting the range of Python tooling it is meant to fit into. It builds on Oxylabs' broader Web Scraper API, which is documented to cover search engines, e-commerce sites, and travel platforms beyond just Google Search.
When to use - and when NOT to
Use it when you need search results or other scraped web content - localized by geography if needed - loaded directly into LlamaIndex documents, without operating your own scraping infrastructure or handling anti-bot measures yourself. Do not use it without an Oxylabs account and credentials already set up; the reader is a thin client over the Oxylabs API and has no scraping capability of its own.
Capabilities
OxylabsGoogleSearchReader.load_data runs a Google search query through Oxylabs, with an optional parse flag for structured output and a geo_location to localize the search, returning results as LlamaIndex documents.
How to install
pip install llama-index-readers-oxylabs
Also installable via poetry add llama-index-readers-oxylabs or uv add llama-index-readers-oxylabs. Requires an Oxylabs account with a username and password.
Who it's for
Developers who want localized search results or other Oxylabs-scraped web data loaded directly into a LlamaIndex pipeline without building their own scraping layer.
Source README
LlamaIndex Oxylabs Reader Integration
Use Oxylabs Reader to load the data from search engines, e-commerce sites, travel platforms, and any other website.
For more information checkout out the Oxylabs documentation.
Instructions for OxylabsReader
Setup and Installation
Installation with pip
pip install llama-index-readers-oxylabs
Installation with poetry
poetry add llama-index-readers-oxylabs
Installation with uv
uv add llama-index-readers-oxylabs
Get Oxylabs credentials
Set up your Oxylabs account and get the username and password.
Using OxylabsReader
from llama_index.readers.oxylabs import OxylabsGoogleSearchReader
reader = OxylabsGoogleSearchReader(
username="OXYLABS_USERNAME",
[REDACTED],
)
docs = reader.load_data(
{"query": "Iphone 16", "parse": True, "geo_location": "Berlin, Germany"}
)
print(docs[0].text)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.