Scrape Websites with Anti-Bot Bypass
Scrape.do Web Scraper retrieves web page content via the Scrape.do API when standard fetch fails with 403, 401, 429, or timeout errors.
Why it matters
Automate the extraction of data from websites, even those with anti-bot measures and dynamic JavaScript rendering, to feed into your data pipelines.
Outcomes
What it gets done
Scrape data from any website using Scrape.do.
Bypass common anti-bot detection mechanisms.
Render JavaScript to access dynamic content.
Extract structured data for further processing.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/scrapedo-web-scraper | bash Overview
Scrape.do Web Scraper
A web scraping tool that uses the Scrape.do API to retrieve page content when standard HTTP requests fail due to blocking errors. Use when standard fetch encounters 403, 401, 429, timeout, access denied, or Cloudflare block errors. Do not use for pages accessible via standard methods or when the API returns quota/token errors.
What it does
Scrape.do Web Scraper retrieves web page content through the Scrape.do API when standard HTTP requests are blocked. The tool extracts either clean text or raw HTML from target pages.
When to use - and when NOT to
Use this skill when standard WebFetch operations fail with blocking errors: 403 Forbidden, 401 Unauthorized, 429 Rate Limit, timeout, access denied messages, or Cloudflare blocks. The source material indicates it should be used automatically when these errors occur.
Do NOT use this for pages that load successfully with standard fetch methods. If the Scrape.do API returns an error (missing token, quota exceeded, or service unavailable), the page cannot be accessed through this method.
Inputs and outputs
You provide a target URL and optionally specify whether you want HTML or text output. The tool accepts URLs via command line or Python function calls.
You receive either the extracted page content (text by default, HTML with the --html flag) on success, or a clear error message explaining why the page is unavailable (missing API token, quota limits, or service issues).
Usage examples
Command line usage:
# Get page text
python scripts/scrape.py https://example.com
# Get HTML
python scripts/scrape.py --html https://example.com
Python integration:
from scripts.scrape import fetch_via_scrapedo
result = fetch_via_scrapedo('https://example.com')
if result['success']:
print(result['content']) # text
# result['html'] - original HTML
else:
print(result['content']) # error description
Who it's for
This skill serves users who need web scraping when standard fetch methods encounter blocking errors. It provides an alternative method for retrieving content from pages that return 403, 401, 429, timeout, access denied, or Cloudflare block errors.
Source README
Scrape.do Web Scraper
Scrape web pages through Scrape.do API. Use when standard fetch fails (blocking, JavaScript).
Usage
### Get page text
python scripts/scrape.py https://example.com
### Get HTML
python scripts/scrape.py --html https://example.com
From Python
from scripts.scrape import fetch_via_scrapedo
result = fetch_via_scrapedo('https://example.com')
if result['success']:
print(result['content']) # text
# result['html'] - original HTML
else:
print(result['content']) # error description
Result
- Success: page text (or HTML with
--html) - Error: clear message (no token / limit exceeded / unavailable)
If an error is returned, the page is inaccessible via this method.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.