Extract Structured Data from Web Pages
Extract structured data and interact with web pages via AgentQL queries or natural language.
Why it matters
Automate the extraction of structured data from any web page, either via REST API or directly from a browser. This tool enables robust web scraping and data retrieval that remains resilient to website changes.
Outcomes
What it gets done
Extract structured data from a given URL using AgentQL queries or natural language prompts.
Extract structured data from the active browser page using AgentQL queries or natural language prompts.
Locate specific web elements within a browser page using natural language descriptions.
Integrate with Playwright for browser automation and interaction.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-tool-tools-agentql | bash Overview
Llama Index Tools Agentql
A LlamaIndex tool that extracts structured web data by URL or live browser page, and locates page elements, using AgentQL queries or natural language. Use REST extraction for simple URL-based data pulls, and the browser tools when the agent is already driving a Playwright session.
What it does
The AgentQL LlamaIndex Tool connects an agent to AgentQL, a service for web interaction and structured data extraction from any web page using either an AgentQL query or a plain natural-language prompt. AgentQL is designed to keep working across multiple languages and pages without breaking as sites change over time.
Three function tools are provided. extract_web_data_with_rest_api extracts structured data as JSON from a web page given a URL, using either an AgentQL query or a natural-language description of what to extract - this works via REST API without needing a live browser. extract_web_data_from_browser extracts structured data as JSON from the currently active page in a browser session, again via an AgentQL query or natural language, but must be used together with a Playwright browser instance. get_web_element_from_browser finds a web element on the active page using a natural-language description and returns its CSS selector, so the agent can then interact with that element - this also requires a Playwright browser.
Only async functions and Playwright browser APIs are supported.
When to use - and when NOT to
Use extract_web_data_with_rest_api when you just need structured data from a URL without an active browser session - the simplest path. Use the browser-based tools (extract_web_data_from_browser, get_web_element_from_browser) when the agent is already driving a Playwright browser and needs to read or interact with the current page state. The source's own worked example chains the two browser tools together: get_web_element_from_browser locates a "next page navigation button" by description and returns its CSS selector, which is then passed straight into the Playwright tool's click method to actually advance the page - a find-then-click pattern useful for paginated or multi-step web tasks. AgentQL's browser tools are meant to be paired with LlamaIndex's own Playwright tools for navigation and clicking. Do not use the browser tools without an active Playwright browser instance - they require one to function, either created fresh via create_async_playwright_browser, or connected to an already-running browser over a Chrome DevTools Protocol (CDP) connection URL using Playwright's own chromium.connect_over_cdp.
Capabilities
extract_web_data_with_rest_api pulls structured JSON from a page by URL. extract_web_data_from_browser pulls structured JSON from the browser's current page. get_web_element_from_browser locates an element by natural-language description and returns a CSS selector for further interaction, such as clicking it via the Playwright tool.
How to install
pip install llama-index-tools-agentql
Also requires configuring the AGENTQL_API_KEY environment variable with a key obtained from the AgentQL Dev Portal.
Who it's for
Developers building LlamaIndex agents that need to browse and extract data from arbitrary web pages reliably - reading structured content, or finding and interacting with page elements - without writing brittle, page-specific scraping code.
Source README
llama-index-tools-agentql
AgentQL provides web interaction and structured data extraction from any web page using an AgentQL query or a Natural Language prompt. AgentQL can be used across multiple languages and web pages without breaking over time and change.
Warning
Only supports async functions and playwright browser APIs, please refer to the following PR for more details: https://github.com/run-llama/llama_index/pull/17808
Installation
pip install llama-index-tools-agentql
You also need to configure the AGENTQL_API_KEY environment variable. You can acquire an API key from our Dev Portal.
Overview
AgentQL provides the following three function tools:
extract_web_data_with_rest_api: Extracts structured data as JSON from a web page given a URL using either an AgentQL query or a Natural Language description of the data.extract_web_data_from_browser: Extracts structured data as JSON from the active web page in a browser using either an AgentQL query or a Natural Language description. This tool must be used with a Playwright browser.get_web_element_from_browser: Finds a web element on the active web page in a browser using a Natural Language description and returns its CSS selector for further interaction. This tool must be used with a Playwright browser.
You can learn more about how to use AgentQL tools in this Jupyter notebook.
Extract data using REST API
from llama_index.tools.agentql import AgentQLRestAPIToolSpec
agentql_rest_api_tool = AgentQLRestAPIToolSpec()
await agentql_rest_api_tool.extract_web_data_with_rest_api(
url="https://www.agentql.com/blog",
query="{ posts[] { title url author date }}",
)
Work with data and web elements using browser
Setup
In order to use the extract_web_data_from_browser and get_web_element_from_browser, you need to have a Playwright browser instance. If you do not have an active instance, you can initiate one using the create_async_playwright_browser utility method from LlamaIndex's Playwright ToolSpec.
Note
AgentQL browser tools are best used along with LlamaIndex's Playwright tools.
from llama_index.tools.playwright.base import PlaywrightToolSpec
async_browser = await PlaywrightToolSpec.create_async_playwright_browser()
You can also use an existing browser instance via Chrome DevTools Protocol (CDP) connection URL:
p = await async_playwright().start()
async_browser = await p.chromium.connect_over_cdp("CDP_CONNECTION_URL")
Extract data from the active browser page
from llama_index.tools.agentql import AgentQLBrowserToolSpec
playwright_tool = PlaywrightToolSpec(async_browser=async_browser)
await playwright_tool.navigate_to("https://www.agentql.com/blog")
agentql_browser_tool = AgentQLBrowserToolSpec(async_browser=async_browser)
await agentql_browser_tool.extract_web_data_from_browser(
prompt="the blog posts with title and url",
)
Find a web element on the active browser page
next_page_button = await agentql_browser_tool.get_web_element_from_browser(
prompt="The next page navigation button",
)
await playwright_tool.click(next_page_button)
Agentic Usage
This tool has a more extensive example for agentic usage documented in this Jupyter notebook.
Run tests
In order to run integration tests, you need to configure L
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.