Scrape Websites with Anti-Bot Bypass

Scrape.do Web Scraper is a Claude Code skill that scrapes web pages through the Scrape.do service when a plain HTTP fetch doesn't work: it bypasses bot-blocking and CAPTCHA challenges, renders JavaScript-heavy pages, and extracts text from the resulting HTML. It's one of 22 skills published in the polyakov-claude-skills marketplace repository.

Use it specifically when a normal fetch fails - the page is blocked, gated behind a CAPTCHA, or only renders its content client-side via JavaScript. That's the skill's own stated trigger condition, and it covers three distinct failure modes an agent otherwise has no way past on its own: anti-bot blocking, an active CAPTCHA challenge, and content that simply doesn't exist in the raw HTML until JavaScript runs. For pages that a plain fetch already handles fine, there's no reason to route through Scrape.do instead - the skill exists to cover the failure case, not to replace fetch as the default.

The skill activates specifically when a plain fetch call fails to return usable content - because the request is blocked, gated behind a CAPTCHA, or the page only renders its content after JavaScript executes - and it hands back the page's text, extracted from the resulting HTML. Its three documented capabilities describe exactly that pipeline: bypassing blocking and CAPTCHA challenges, rendering JavaScript-heavy pages, and extracting text from the HTML that results, so the caller gets readable content instead of an unusable response.

Install it via the marketplace: /plugin marketplace add artwist-polyakov/polyakov-claude-skills followed by /plugin install scrapedo-web-scraper. Without the marketplace, copy the skill folder directly into .claude/skills/ - globally at ~/.claude/skills/ for use across every project, or into a single project's own .claude/skills/ directory - and Claude Code picks it up on the next run. A plugin can also be loaded for local testing with claude --plugin-dir ./plugins/<plugin-name>. It ships inside the same polyakov-claude-skills marketplace as 21 other independently installable skills - among them a SEO site crawler (crawl4ai-seo), a Word-template filler (docx-contracts), search-research tools (perplexity-search, x-research), and integrations for Yandex, Telegram, and Reddit - each installed the same way and entirely separate from this skill's own scraping functionality. The repository is MIT licensed.

Anyone running Claude Code who needs a scraping fallback for pages a plain fetch can't handle - sites behind bot protection, CAPTCHA, or JavaScript-only rendering. It's most useful inside research or content-gathering workflows already built on Claude Code skills, where a single blocked or JS-only page would otherwise stall the whole task instead of just that one fetch.