Extract Structured Data from Websites

Web Scraper is a multi-strategy skill for extracting structured data (tables, lists, prices, contacts, and more) from web pages, with pagination handling, monitoring, and CSV/JSON export. It runs a strict 7-phase pipeline - Clarify -> Recon -> Strategy -> Extract -> Transform -> Validate -> Format - never skipping Clarify or Recon except on a fast path (a single simple page with a clear URL and data target compresses phases 1-3 into one WebFetch call). Five extraction strategies are selected by a decision tree during Phase 3: Strategy A (WebFetch with an AI extraction prompt, for static pages, auto-escalating to Strategy B if fewer than 50% of the expected items come back), Strategy B (Browser automation - tabs_context_mcp, navigate, computer for waits/scrolls/clicks, find for elements, read_page/get_page_text, and javascript_tool with table/list-extraction JS snippets - for JS-rendered or SPA pages), Strategy C (Bash curl + jq for JSON/XML APIs and CSV downloads), Strategy D (a hybrid combining WebSearch discovery, WebFetch assessment, Browser for JS-heavy sections, and Bash post-processing), and Strategy E (extracting embedded JSON-LD/microdata directly via javascript_tool, preferred when it covers the needed fields since it's cleaner than DOM scraping). Ten extraction modes are supported - table, list, article, product, contact, FAQ, pricing, events, jobs, and custom - each with its own tailored WebFetch prompt and field list, e.g. product mode extracts name/brand/price/currency/rating/specifications, pricing mode extracts planName/monthlyPrice/features/highlighted.

Use it whenever a user asks to scrape, extract structured web data, or collect data from a site - including discovery mode, where a topic without a specific URL triggers a WebSearch for the top 3-5 candidate pages before the user picks one. It is not the right skill for tasks unrelated to web scraping or where a simpler, more specific tool already fits. Hard access-respect rules govern every run: never bypass bot detection, CAPTCHAs, or access controls; never scrape behind authentication without the user's explicit access; respect robots.txt; rate-limit to one request per 2 seconds on sequential fetches and stop immediately on a 429 response; never reproduce large blocks of copyrighted article text (summarize narrative content instead, always with source attribution); extract only what was explicitly requested and warn before collecting PII like emails or phone numbers at scale. On failure or a block, it explains the specific reason, suggests alternatives, and never retries aggressively.

Phase 1 resolves required parameters (target URL(s), the specific data target, output format - defaulting to a Markdown table - and scope) plus optional ones (pagination and max pages, max items, filters, sort order, save path, language, diff mode), asking only the minimum clarifying question when the request is ambiguous. Phase 2 recon uses WebFetch to report page type, content structure, item count, JS-rendering indicators, pagination signals, data density, available fields, and embedded structured data, which Phase 3 uses to pick a strategy. Pagination is handled per mechanism: page-number pagination iterates a detected URL pattern up to a default 5-page cap; infinite scroll extracts, scrolls, and re-extracts, stopping after 2 scrolls with no new items; a "Load More" button is clicked and re-extracted until it disappears. Phase 5 always applies whitespace cleanup, HTML-entity decoding, Unicode normalization, and empty-string-to-null conversion, and conditionally applies price/date/URL/phone normalization, deduplication, and sorting; when useful it also enriches records with currency conversion, a domain column split out of URLs, an article word count, or a "days ago" relative-date column. Deduplication keeps the first row on an exact match across all fields, and on a near match (same key fields but differing details) keeps whichever row is most complete and flags the merge in the delivery notes. Phase 6 validates item count, empty-field rate, type consistency, duplicates, encoding, completeness, truncation, and outliers, then assigns a HIGH/MEDIUM/LOW confidence rating with specifics (e.g. "HIGH - 47 items, all 6 fields populated"), attempting auto-recovery - retry with Browser, re-decode encoding, check for missed pagination - before reporting an issue. Phase 7 wraps every delivery in a metadata envelope (source, date, item/field counts, confidence, strategy used, format) and formats per output: Markdown tables use aligned columns and N/A for gaps, JSON uses camelCase keys inside a metadata/data envelope, and CSV uses UTF-8 with a BOM and quotes fields containing commas. Diff mode marks [NEW], [REMOVED], and [WAS: old_value] against the previous run.

Three bundled reference files extend the skill: references/extraction-patterns.md for CSS selectors and JavaScript snippets, references/output-templates.md for complete Markdown/JSON/CSV templates, and references/data-transforms.md for cleaning and enrichment patterns. Multi-URL jobs extract a schema from the first URL, confirm it with the user, then apply it to the remaining URLs, tagging every record with its source column.

Users who need structured data pulled from web pages - tables, listings, prices, contacts, FAQs, jobs, or events - with automatic strategy selection between static fetching, browser automation, and API calls, plus built-in pagination, validation, and export to Markdown, JSON, or CSV.