Extract Public Web Data via API

Provides one unified interface to HasData's cloud web-data-extraction platform - all endpoints under https://api.hasdata.com, authenticated with a single x-api-key header - across three execution modes: the Web Scraping API (POST /scrape/web, seconds-latency, JS rendering, CSS/AI extraction, screenshots) for arbitrary URLs; synchronous Scraper APIs (GET /scrape/<vertical>/<resource>) returning pre-parsed JSON for known platforms across ten verticals (Google Search/SERP/AI Mode/News/Shopping/Bing, Amazon and Shopify ecommerce, Zillow/Redfin real estate, Airbnb/Booking/Google Flights travel, Google Maps/Yelp/YellowPages local business, Indeed/Glassdoor jobs, YouTube); and async Scraper Jobs (POST /scrapers/<slug>/jobs) for bulk extraction, recursive crawling, and webhook fan-out via named jobs like crawler, contacts, sec-edgar, amazon-bestsellers, and amazon-product-reviews. Its own decision rule: default to a Scraper API when one exists for the platform since it needs no selector maintenance, fall back to Web Scraping for URLs no API covers, and reach for a Scraper Job only when no API equivalent exists or async fan-out with webhooks beats a paginated client loop.

curl -G 'https://api.hasdata.com/scrape/google/serp' \
  --data-urlencode 'q=coffee' \
  -H 'x-api-key: <your-api-key>'

Use it for web scraping, search-engine results, or structured data extraction across ecommerce, travel, jobs, or local-business platforms. Every response carries the same envelope (requestMetadata: {id, status, url} plus endpoint-specific data), and requestMetadata.status === "ok" is the only real success signal - an HTTP 200 alone isn't enough. Five named gotchas govern correct use: the server enforces a 300-second deadline, so the client timeout must be set to at least that or requests will appear to fail while still being billed on completion; jsRendering should stay disabled first and only be enabled for pages that actually need a headless browser, since most static pages parse fine without it; cookies go through headers["Cookie"], there is no separate cookies parameter; includePaths regex matching is case-sensitive; and async Scraper Job results are double-wrapped (body.data[i].data, with id/jobId/dataId/timestamps on the outer object). Its limitations section is explicit: use only for public data or content the requester is authorized to access, respecting site terms, robots and access controls, privacy law, and rate limits - flagged specifically for lead generation from Maps-derived contact details and personal email/phone lookups, which require a legitimate purpose and authorization.

Inputs vary by mode: a target URL plus optional extractRules (CSS selectors) or aiExtractRules (natural-language extraction) and JS-interaction scenarios for Web Scraping; a query or identifier (search term, ASIN, listing URL) for Scraper APIs; and a job configuration, such as outputFormat: ["markdown"] and includePaths, for Scraper Jobs. Output is JSON in all cases, with Scraper Jobs additionally offering short-lived download URLs for csv/json/xlsx once the job status reaches finished. The async submit response's handle is body.id, an integer, explicitly not jobId, which must be persisted immediately and polled via GET /scrapers/jobs/<id> every 10-30 seconds with backoff, treating webhooks (delivered best-effort with 3 retries) as a supplement to polling, never a replacement.

Five named high-leverage patterns chain HasData calls into larger workflows: SERP-first enrichment for company or professional lookups before any direct scraping; "AI Mode + verify" (/scrape/google/ai-mode for an answer plus references, then /scrape/web in markdown on each reference URL to build cited RAG context with no vector database); Maps-to-leads (/scrape/google-maps/search for business site and phone data); a crawler Scraper Job with a markdown output format and a path filter to produce an LLM-ready corpus in one submission; and reading SERP rich snippets (knowledgeGraph, localResults, inlineShoppingResults, relatedQuestions) before falling back to direct page access, since they're already pre-parsed. A bundled references/code-recipes.md supplies ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full async job lifecycle.

Developers building a data-extraction, lead-gen, price-monitoring, or LLM-corpus pipeline who want one API key and a consistent response shape across search, ecommerce, travel, real estate, local-business, and job-listing sources, rather than integrating a separate scraper per platform.