Integrate Google Search Results with LlamaIndex
Get top Google search result URLs into LlamaIndex documents via the Zyte SERP API.
Why it matters
Leverage Zyte's Google Search API integration to enrich your LlamaIndex applications with real-time organic search results. This asset allows you to programmatically fetch top search result URLs based on a given query, enabling more comprehensive data ingestion for your AI.
Outcomes
What it gets done
Fetch Google search result URLs using Zyte API
Integrate Zyte's search capabilities into LlamaIndex pipelines
Extract relevant content from fetched URLs using ZyteWebReader
Build RAG systems with up-to-date web data
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-zyte-serp | bash Overview
LlamaIndex Readers Integration: Zyte-Serp
A LlamaIndex reader that returns top Google search result URLs via Zyte, pairable with ZyteWebReader for full article content. Use ZyteSerpReader alone for just URLs, or pair with ZyteWebReader when the actual article content is needed.
What it does
ZyteSerp adds organic search results from Google Search into LlamaIndex, taking a query and returning the top search result URLs as documents. ZyteSerpReader is initialized with a Zyte API key and an extract_from option choosing how extraction happens - either httpResponseBody or browserHtml. load_data then takes the search query and returns the matching result URLs.
The source's own worked example goes a step further than just collecting URLs: it takes the URLs returned by ZyteSerpReader, then passes them to a separate reader, ZyteWebReader, initialized in "article" mode, which fetches the actual page content from those URLs - extracting just the article text rather than full raw HTML. This two-step pattern (search first, then fetch content from the results) turns a plain search into a way to pull real, current web content grounded in a live Google search.
When to use - and when NOT to
Use ZyteSerpReader alone when you only need the list of top result URLs for a query, without their content. Pair it with ZyteWebReader in "article" mode when you need the actual article content from those search results, not just the links. Do not use it without a Zyte API key from Zyte's API service; both readers depend on that key to function.
Capabilities
ZyteSerpReader.load_data returns the top Google search result URLs for a query. Paired with ZyteWebReader (mode "article"), the URLs can be fetched and reduced to their article content.
How to install
pip install llama-index-readers-zyte-serp
Requires a Zyte API key.
Who it's for
Developers who need current Google search results, and optionally the article content behind those results, loaded into LlamaIndex.
Source README
LlamaIndex Readers Integration: Zyte-Serp
ZyteSerp can be used to add organic search results from Google Search. It takes a query and returns top search results urls.
Instructions for ZyteSerpReader
Setup and Installation
pip install llama-index-readers-zyte-serp
Secure an API key from Zyte to access the Zyte services.
Using ZyteSerpReader
Initialization: Initialize the ZyteSerpReader by providing the API key and the option for extraction ("httpResponseBody" or "browserHtml").
from llama_index.readers.zyte_serp import ZyteSerpReader zyte_serp = ZyteSerpReader( api_key="your_api_key_here", extract_from="httpResponseBody", # or "browserHtml" )Loading Data: To load search results, use the
load_datamethod with the query you wish to search.
documents = zyte_serp.load_data(query="llama index docs")
Example Usage
Here is an example demonstrating how to initialize the ZyteSerpReader and get top search URLs.
Further the content from these URLs can be loaded using ZyteWebReader in "article" mode to obtain just the article content from webpage.
from llama_index.readers.zyte_serp import ZyteSerpReader
from llama_index.readers.web.zyte.base import ZyteWebReader
### Initialize the ZyteSerpReader with your API key
zyte_serp = ZyteSerpReader(
api_key="your_api_key_here", # Replace with your actual API key
)
### Get the search results (URLs from google search results)
search_urls = zyte_serp.load_data(query="llama index docs")
### Display the results
print(search_urls)
urls = [result.text for result in search_urls]
### Initialize the ZyteWebReader to load the content from search results
zyte_web = ZyteWebReader(
api_key="your_api_key_here", # Replace with your actual API key
mode="article",
)
documents = zyte_web.load_data(urls)
print(documents)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.