Fetch and Extract Web Content with Bot Protection
Fetch content from bot-protected websites for Claude - escalating stealth levels bypass anti-automation, returns clean HTML/Markdown.
Why it matters
Access and extract text content from websites, even those with bot protection, to bridge the gap between browser views and AI accessibility.
Outcomes
What it gets done
Fetch complete web pages with pagination support.
Extract specific content using regex patterns.
Bypass bot protection with multiple stealth modes.
Retrieve documentation and reference materials.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/vb-scrapling-fetch | bash Capabilities
Tools your agent gets
Fetches complete web pages with pagination support and bot protection bypass
Finds and extracts specific content from pages using regex patterns
Overview
scrapling-fetch MCP Server
scrapling-fetch-mcp helps Claude fetch content from bot-protected websites via escalating stealth levels (basic, stealth, max-stealth), returning clean HTML or Markdown, optimized for low-volume documentation retrieval. Use it for low-volume retrieval of documentation or reference content from bot-protected sites. Not designed for high-volume scraping, data harvesting, or authenticated sites.
What it does
scrapling-fetch-mcp helps Claude fetch content from bot-protected websites, using Scrapling (patchright + curl-cffi) to bypass anti-automation measures and return clean HTML or Markdown. It's optimized for low-volume retrieval of documentation and reference material, not high-volume scraping or bulk data harvesting.
When to use - and when NOT to
Use it when a site you want Claude to read blocks typical automated fetches - documentation sites, reference pages, or articles protected against bots. It escalates automatically through three protection levels: basic (fast, 1-2s, works for most sites), stealth (moderate, 3-8s, headless Chromium), and max-stealth (thorough, 10s+, full browser fingerprint). Do not use it for high-volume scraping, data harvesting, or sites requiring authentication - it's explicitly scoped to low-volume, text-content retrieval (documentation, articles, references), and performance varies significantly by site complexity and protection level.
Capabilities
- Page fetching: retrieve complete pages with pagination support.
- Pattern extraction: find content matching a regex pattern within a fetched page.
Both tools are used automatically by Claude once installed - you just ask naturally, e.g. "Fetch the docs at https://example.com/api" or "Find all mentions of 'authentication' on that page."
How to install
Requires Python 3.10+ and uv. Easiest as a Claude Code skill - install into your project:
git clone --depth=1 https://github.com/cyberchitta/scrapling-fetch-mcp /tmp/scrapling-fetch-mcp
cp -r /tmp/scrapling-fetch-mcp/skills/s-fetch .claude/skills/
rm -rf /tmp/scrapling-fetch-mcp
(Or copy to ~/.claude/skills/ for all projects.) The first /s-fetch invocation triggers one-time setup, installing the tool and downloading browser binaries.
For Claude Desktop as an MCP server:
uv tool install git+https://github.com/cyberchitta/scrapling-fetch-mcp
uvx --from git+https://github.com/cyberchitta/scrapling-fetch-mcp scrapling install
{
"mcpServers": {
"scrapling-fetch": {
"command": "uvx",
"args": ["scrapling-fetch-mcp"]
}
}
}
Browser binary download is several hundred MB and must finish before first use - if the server times out initially, wait a few minutes and retry.
Who it's for
Developers who need Claude to read documentation or reference pages on sites that block ordinary automated requests, without building custom bypass logic themselves.
Source README
scrapling-fetch-mcp
Helps AI assistants fetch content from bot-protected websites. Uses Scrapling (patchright + curl-cffi) to bypass anti-automation measures, returning clean HTML or Markdown.
Optimized for low-volume retrieval of documentation and reference materials. Not designed for high-volume scraping or data harvesting.
Requirements: Python 3.10+, uv
Claude Code Skill
The easiest way to use this is as a Claude Code skill. Once installed, Claude will automatically fetch bot-protected URLs when you ask - no manual commands needed.
Install into your project (recommended - only loads in this project's context):
git clone --depth=1 https://github.com/cyberchitta/scrapling-fetch-mcp /tmp/scrapling-fetch-mcp
cp -r /tmp/scrapling-fetch-mcp/skills/s-fetch .claude/skills/
rm -rf /tmp/scrapling-fetch-mcp
Or install for all projects (loads into context everywhere):
git clone --depth=1 https://github.com/cyberchitta/scrapling-fetch-mcp /tmp/scrapling-fetch-mcp
cp -r /tmp/scrapling-fetch-mcp/skills/s-fetch ~/.claude/skills/
rm -rf /tmp/scrapling-fetch-mcp
The first time you invoke /s-fetch, Claude will readskills/s-fetch/references/install.md and run the one-time setup
(installs the tool and downloads browser binaries). After that, just
ask naturally:
"Fetch the docs at https://example.com/api"
"Find all mentions of 'authentication' on that page"
"Get me the installation instructions from their homepage"
Claude Desktop (MCP Server)
If the /s-fetch skill has already installed the tool, skip to the config below.
Otherwise install first:
uv tool install git+https://github.com/cyberchitta/scrapling-fetch-mcp
uvx --from git+https://github.com/cyberchitta/scrapling-fetch-mcp scrapling install
Note: Browser installation downloads hundreds of MB and must complete before first use. If the server times out initially, wait a few minutes and try again.
Add this to your Claude Desktop MCP settings and restart:
MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
{
"mcpServers": {
"scrapling-fetch": {
"command": "uvx",
"args": ["scrapling-fetch-mcp"]
}
}
}
How It Works
Two tools, used automatically by Claude:
- Page fetching - retrieves complete pages with pagination support
- Pattern extraction - finds content matching a regex
Three protection levels, escalated automatically:
- basic - fast (1-2s), works for most sites
- stealth - moderate (3-8s), headless Chromium
- max-stealth - thorough (10s+), full browser fingerprint
Limitations
- Text content only (documentation, articles, references)
- Not for high-volume scraping or sites requiring authentication
- Performance varies by site complexity and protection level
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.