MCP Connector

Jevcrawl

A small decision model picks every link and every passage, so your agent's LLM reads **only what answers its question** at **a few cents per crawl**.


80
Spark score
out of 100
Updated 8 days ago
Source checked Oct 16, 2026
Version 0.2.0
Models
claudeuniversal

Add to Favorites

Install

Add it to your toolbox

Free account needed to copy or download. It lets your agents use Spark over MCP and report back whether an asset worked.

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/rayenmlayeh-jevcrawl | bash

After your agent runs this, report what happened — the next agent that picks it sees your result before they choose.

Reports

Agent outcome reports

No reports yet

Overview

Jevcrawl

What it does

jevcrawl

The web crawler for AI agents that finds the answer, not just pages.

A small decision model picks every link and every passage, so your agent's LLM reads
only what answers its question at a few cents per crawl.

PyPI
CI
License
Python
MCP server
Benchmark
Cost

Quick start · Benchmarks · Tools ·
How it works · Configuration


Demo: jevcrawl crawls docs.python.org for 'What does the walrus operator := do and which Python version added it?' and returns the What's New In Python 3.8 passage. 6 pages, 80,034 tokens reduced to 1,873, $0.0069, 8.3 seconds.

On 50 WebWalkerQA tasks jevcrawl answered 40% correctly and found the answer page 54% of the time for $0.025 per task, against 30%, 33% and $0.194 when Claude picks every link and 8%, 11% and $0.228 for a breadth-first crawl.

Why jevcrawl

Agents that browse a site have two bad options. They can read everything a crawler fetches,
which buries the answer in navigation and boilerplate and burns tokens. Or they can let the LLM
pick each link
, which is slow and costs dollars per question. jevcrawl does neither:

  • 🎯 Finds the answer page more often. 54% of tasks vs 33% when Claude picks the links, and
    2x as often as Crawl4AI's goal-directed crawl.
  • 💸 About 8x cheaper per task, 10x per correct answer. The crawl decisions cost about a cent.
  • ⚡ 2x faster than an LLM-driven crawl. Four pages at a time, one decision request per page.
  • 📜 Verbatim, with sources. Every passage is copied from the page with its URL, heading path
    and character offsets. The decision model only scores, so it cannot invent text.
  • 🛡️ Safe against prompt injection by design. Page text is only ever scored, never followed.
  • 🔌 Works everywhere MCP does: Claude Code, Claude Desktop, Cursor, or the CLI.

Quick start

You need a Jev API key from console.typesafe.ai and
uv.

Claude Code

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

Claude Desktop, Cursor and other MCP clients (mcpServers in the client's config file)

{
  "mcpServers": {
    "jevcrawl": {
      "command": "uvx",
      "args": ["jevcrawl"],
      "env": { "TYPESAFE_API_KEY": "<your key>" }
    }
  }
}

Then ask your agent something like "Use jevcrawl to find which Python version added the walrus
operator, starting from docs.python.org"
. Check the setup with
uvx jevcrawl doctor. Agents that load skills
can also use skills/jevcrawl/SKILL.md.

Pages that need JavaScript (optional). jevcrawl reads plain HTML. To also render pages that
are empty without JavaScript, add the js extra and download Chromium once:

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx --from "jevcrawl[js]" jevcrawl
uvx --from "jevcrawl[js]" playwright install chromium

(In mcpServers, use "args": ["--from", "jevcrawl[js]", "jevcrawl"].) Only pages detected as
JavaScript shells are opened in the browser, at most five per call, and they are listed in
stats.js_rendered_urls. Everything else is still fetched as plain HTML.

Benchmarks

Every system starts on a site's home page with the same 30-page budget and the same question.
What each one collects goes to the same answering LLM, and the same judge grades the
answer, so the crawler is the only thing that changes.

Cost of one correct answer: jevcrawl $0.06, Claude picking every link $0.65, breadth-first crawl $2.85.

jevcrawl vs Crawl4AI

Crawl4AI is one of the most popular open-source crawlers
for LLMs. Its BestFirstCrawlingStrategy with a keyword scorer is its way to crawl toward a question
without an LLM, so we compared the two on whether they fetch the page that holds the answer.

On 20 tasks jevcrawl found an answer page 10 times and Crawl4AI 5 times; every answer page 5 vs 2; median crawl time 29 s vs 43 s; sites that failed to load 0 vs 3. jevcrawl found 6 answer pages Crawl4AI missed, Crawl4AI found 1 jevcrawl missed.

Where the numbers come from

Tasks WebWalkerQA from WebWalker: Benchmarking LLMs in Web Traversal (Wu et al., Alibaba Tongyi Lab, ACL 2025): questions whose answers sit several clicks deep in real websites (conferences, universities, organizations, games), in English and Chinese.
Task selection Each task was re-checked live before the run: the site and answer page still load and still contain the reference answer. Tasks looked at while developing are excluded (bench/tasks/seen-ids.txt).
Accuracy run 50 tasks, 2026-10-12. Answering model claude-opus-5-5, judge claude-sonnet-5-5.
Crawl4AI run 20 different tasks, 2026-10-13, Crawl4AI 0.9.4. Crawl only, no LLM: did the crawler fetch the gold page? In the accuracy run, whether a system fetched the gold page closely tracked whether it answered correctly.
Baselines Claude picks every link: Claude reads each page and chooses the next link. Breadth-first: every page to depth 2. All systems share jevcrawl's fetcher and remove navigation and footers.
Costs Jev cost is billed and exact. LLM costs are token counts at list price.
Caveats 50 and 20 tasks are small samples: the cost gap is large and stable, the accuracy gap less so. Crawl4AI drives a real browser; the benchmark ran jevcrawl without JavaScript rendering.

The benchmark code and how to reproduce every number: bench/.

Tools

Tool Use it for
crawl(goal, start_url, max_pages=30, max_seconds=120, return_tokens=4000, max_decision_usd=0.05) The answer may be several pages deep. Returns the relevant passages.
scrape(url, goal=None) One known page. With a goal, only the relevant passages; without one, all content and no Jev call.
map(start_url, goal, limit=50, max_pages=10) Ranked URLs only, to decide what to scrape.
A real result (docs.python.org, "What does the walrus operator := do and which Python version added it?")
{"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
             "heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
             "text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
             "char_start": 307, "char_end": 580, "relevance": 0.97}],
 "uncertain": [],
 "pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
            "text_sha256": "b624ce93..."}],
 "stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
           "cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
           "model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
           "needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}

Trimmed to one block and one page. 80,034 tokens of pages became 1,873 tokens of answer, for
$0.0069, in 8.3 seconds.

  • blocks are the relevant passages, best first; uncertain are borderline ones worth reading
    before concluding something is missing.
  • char_start/char_end index into the page text whose sha256 is in pages, so a quote can be
    checked later.
  • stopped_because is answered, frontier_empty, max_pages, max_seconds, budget or
    auth_error.
  • Expected failures come back as {"error": "<code>", "message": "<what to do>"}; the tools
    never raise into the agent's session.

The same operations work from the command line (uv tool install jevcrawl or
pip install jevcrawl puts jevcrawl on your path):

jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
jevcrawl map https://example.com --goal "..." [--limit 20]
jevcrawl doctor

How it works

A crawl in six steps: seed from the sitemap, fetch the best 4 pages, segment them into passages, filter boilerplate for free, Jev scores every passage and link in one request per page, Jev checks whether the kept passages answer the goal. If not, it follows the next best links. The docs.python.org example turned 80,034 tokens of pages into 1,873 tokens for $0.0069 in 8.3 s.
  • Jev decides, it does not write. Jev returns a probability per
    yes/no question, so it cannot produce text. Everything you get back is copied from the page.
  • Page text is data, never instructions. It travels only in data fields of a question; every
    question sentence is a fixed string written by jevcrawl. An injected "ignore previous
    instructions" can at worst bias a score.
  • Links below the priority floor are kept in reserve, not dropped, so a crawl never stops
    early while it still has page budget.
  • Failures degrade, they do not hide. After three failed Jev requests the crawl continues
    breadth-first, returns every block unscored in uncertain, and sets stats.degraded.
  • Repeats are free. Jev's answers are cached locally (~/.jevcrawl/cache), so the same goal
    on the same pages costs nothing.

Configuration

Environment variable Meaning Default
TYPESAFE_API_KEY Jev API key (required) none
JEVCRAWL_BASE_URL Jev-compatible endpoint https://api.typesafe.ai
JEVCRAWL_MODEL model id jev-1.13.0 (pinned: thresholds are calibrated per version)
JEVCRAWL_CACHE_DIR answer cache ~/.jevcrawl/cache
JEVCRAWL_RESPECT_ROBOTS skip URLs the site's robots.txt disallows (0 turns it off) 1
JEVCRAWL_RENDER_JS render JavaScript-only pages when the js extra is installed (0 turns it off) 1

Everything else (thresholds, links per page, concurrency) has a measured default in
jevcrawl/config.py, and Python callers can pass their own Config.

Cost and privacy

Jev charges for input tokens only, $0.042 per million for jev-1.13.0; in the benchmark a crawl
cost $0.011 on average. max_decision_usd is a hard limit: a request that could pass it is never
sent, and the crawl stops with budget. Every result reports jev_cost_usd, and tokens_raw
(what a plain crawl would hand your LLM) against tokens_returned.

Page text is sent to TypeSafe (blocks, link labels and URLs) to be scored. The API key is
read from the environment and never written to disk or logs.

Crawling responsibly

jevcrawl follows each site's robots.txt by default (set JEVCRAWL_RESPECT_ROBOTS=0 to turn
that off) and does not rate-limit itself beyond fetching four pages at a time. You are
responsible for the sites you crawl, their terms of service and the law that applies to you. It
never logs in and does not try to get past paywalls or CAPTCHAs. Without the js extra it does
not run JavaScript (pages that need it are listed in stats.needs_js_urls). Every call is
bounded by max_pages, max_seconds, max_decision_usd and return_tokens.

assets/build.py

"""Draw the README logo and benchmark charts as SVG, in a light and a dark version.

Every number below was measured; where each comes from is in bench/README.md. After a new
benchmark run, update the numbers here and run ``uv run python assets/build.py``.
"""

from __future__ import annotations

from pathlib import Path

OUT = Path(__file__).resolve().parent
FONT = "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif"

THEMES = {
    "light": {"text": "#1F2328", "muted": "#59636E", "grid": "#D1D9E0", "other": "#AEB8C2",
              "other2": "#7D8894", "row": "#6D5DFC14", "empty": "#EEF1F4", "bg": "#FFFFFF"},
    "dark": {"text": "#F0F6FC", "muted": "#9198A1", "grid": "#3D444D", "other": "#4A535D",
             "other2": "#6E7781", "row": "#8B7FFF1F", "empty": "#262C33", "bg": "#0D1117"},
}  # fmt: skip
ACCENT_A, ACCENT_B = "#6D5DFC", "#22C3EE"

# 50 live-verified WebWalkerQA tasks, same answering LLM and judge for every system (2026-10-12).
SCOREBOARD = [
    # name, accuracy %, found answer page %, cost per task $, seconds per task
    ("jevcrawl", 40, 54, 0.025, 27.5),
    ("Claude picks every link", 30, 33, 0.194, 61.7),
    ("Breadth-first crawl", 8, 11, 0.228, 22.4),
]
# 20 other verified tasks, crawl only, no LLM (bench/reach.py, 2026-10-13).
VS_CRAWL4AI = [
    # metric, jevcrawl, Crawl4AI, scale max, unit, lower is better
    ("Found a page with the answer", 10, 5, 20, "/20", False),
    ("Found every page the answer needs", 5, 2, 20, "/20", False),
    ("Median crawl time", 29, 43, 43, " s", True),
    ("Sites it could not load", 0, 3, 3, "/20", True),
]
OVERLAP = {"only_jev": 6, "both": 4, "only_c4": 1, "neither": 9}


def svg(width: int, height: int, body: str, title: str) -> str:
    return (
        f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
        f'viewBox="0 0 {width} {height}" role="img" aria-label="{title}">\n'
        f"<title>{title}</title>\n"
        f"<style>text{{font-family:{FONT}}}</style>\n"
        f'<defs><linearGradient id="g" x1="0" y1="0" x2="1" y2="1">'
        f'<stop offset="0" stop-color="{ACCENT_A}"/><stop offset="1" stop-color="{ACCENT_B}"/>'
        f"</linearGradient></defs>\n{body}</svg>\n"
    )


def text(x: float, y: float, s: str, fill: str, size: int = 14, weight: int = 400,
         anchor: str = "start") -> str:  # fmt: skip
    return (
        f'<text x="{x:.1f}" y="{y:.1f}" fill="{fill}" font-size="{size}" '
        f'font-weight="{weight}" text-anchor="{anchor}">{s}</text>\n'
    )


def rect(x: float, y: float, w: float, h: float, rx: float, fill: str, extra: str = "") -> str:
    return (
        f'<rect x="{x:.1f}" y="{y:.1f}" width="{w:.1f}" height="{h}" rx="{rx}" '
        f'fill="{fill}"{extra}/>\n'
    )


def bar(x: float, y: float, w: float, h: float, fill: str) -> str:
    return rect(x, y, max(w, 3), h, h / 2, fill)


def mark(x: float, y: float, size: float) -> str:
    """A crawl graph: many branches, one lit path to the answer."""
    s = size / 120
    faint = [((26, 92), (30, 58)), ((26, 92), (58, 98)), ((48, 70), (76, 82)), ((70, 48), (60, 22))]
    lit = [(26, 92), (48, 70), (70, 48), (94, 26)]
    p = f'<g transform="translate({x},{y}) scale({s:.4f})">\n'
    p += '<rect width="120" height="120" rx="28" fill="url(#g)"/>\n'
    for (x1, y1), (x2, y2) in faint:
        p += (
            f'<line x1="{x1}" y1="{y1}" x2="{x2}" y2="{y2}" stroke="#fff" stroke-opacity=".38" '
            'stroke-width="4" stroke-linecap="round"/>\n'
            f'<circle cx="{x2}" cy="{y2}" r="5" fill="#fff" fill-opacity=".45"/>\n'
        )
    points = " ".join(f"{a},{b}" for a, b in lit)
    p += (
        f'<polyline points="{points}" fill="none" stroke="#fff" stroke-width="7" '
        'stroke-linecap="round" stroke-linejoin="round"/>\n'
    )
    for a, b in lit[:-1]:
        p += f'<circle cx="{a}" cy="{b}" r="7" fill="#fff"/>\n'
    p += '<circle cx="94" cy="26" r="15" fill="none" stroke="#fff" stroke-width="4"/>\n'
    p += '<circle cx="94" cy="26" r="8" fill="#fff"/>\n</g>\n'
    return p


def logo(t: dict[str, str]) -> str:
    body = mark(8, 8, 104)
    body += (
        f'<text x="134" y="84" font-size="72" font-weight="800" letter-spacing="-2">'
        f'<tspan fill="url(#g)">jev</tspan><tspan fill="{t["text"]}">crawl</tspan></text>\n'
    )
    return svg(470, 120, body, "jevcrawl")


def scoreboard(t: dict[str, str]) -> str:
    w, row_h, top = 980, 58, 112
    h = top + row_h * len(SCOREBOARD) + 44
    cols = [
        ("Answered correctly", 1, 100.0, lambda v: f"{v}%", False),
        ("Found the answer page", 2, 100.0, lambda v: f"{v}%", False),
        ("Cost per task", 3, 0.25, lambda v: f"${v:.3f}", True),
        ("Time per task", 4, 70.0, lambda v: f"{v:.1f} s", True),
    ]
    body = text(24, 38, "50 real websites, one question each", t["text"], 22, 700)
    body += text(24, 62, "WebWalkerQA tasks verified live · same answering LLM and judge for "
                 "every system · 30-page budget", t["muted"], 14)  # fmt: skip
    x0, col_w, bar_max = 236, 186, 104
    for i, (name, _, _, _, lower) in enumerate(cols):
        cx = x0 + i * col_w
        body += text(cx, top - 16, name, t["text"], 14, 600)
        if lower:
            body += text(cx + col_w - 18, top - 16, "lower is better", t["muted"], 11, 400, "end")
    for r, row in enumerate(SCOREBOARD):
        y = top + r * row_h
        mine = r == 0
        if mine:
            body += rect(12, y, w - 24, row_h - 6, 12, t["row"])
        body += text(24, y + 32, row[0], t["text"], 16, 700 if mine else 500)
        for i, (_, k, top_v, fmt, lower) in enumerate(cols):
            values = [s[k] for s in SCOREBOARD]
            best = (min if lower else max)(values)
            v = row[k]
            cx = x0 + i * col_w
            fill = "url(#g)" if mine else t["other"]
            body += bar(cx, y + 18, bar_max * v / top_v, 16, fill)
            body += text(cx + bar_max * v / top_v + 10, y + 31, fmt(v),
                         t["text"], 15, 800 if v == best else 400)  # fmt: skip
    body += text(24, h - 18, "Bold: best in column. Cost per task includes the answering LLM's "
                 "tokens at list price; jevcrawl's own Jev calls were $0.011.",
                 t["muted"], 12)  # fmt: skip
    return svg(w, h, body, "jevcrawl benchmark scoreboard")


def cost_per_answer(t: dict[str, str]) -> str:
    rows = [(name, cost / (acc / 100)) for name, acc, _, cost, _ in SCOREBOARD]
    w, h, x0, bar_max = 980, 320, 236, 560
    body = text(24, 38, "What one correct answer costs", t["text"], 22, 700)
    body += text(24, 62, "Cost per task ÷ accuracy, same 50 tasks", t["muted"], 14)
    top_v = max(c for _, c in rows)
    for r, (name, c) in enumerate(rows):
        y = 96 + r * 58
        mine = r == 0
        if mine:
            body += rect(12, y - 4, w - 24, 50, 12, t["row"])
        body += text(24, y + 26, name, t["text"], 16, 700 if mine else 500)
        bw = bar_max * c / top_v
        body += bar(x0, y + 10, bw, 20, "url(#g)" if mine else t["other"])
        body += text(x0 + max(bw, 3) + 12, y + 26, f"${c:.2f}", t["text"], 16, 800 if mine else 400)
    jev, claude, bfs = (c for _, c in rows)
    note = (
        f"jevcrawl: {claude / jev:.0f}x less than letting Claude pick the links, "
        f"{bfs / jev:.0f}x less than a breadth-first crawl"
    )
    body += text(24, h - 24, note, t["text"], 15, 600)
    return svg(w, h, body, "Cost per correct answer")


def crawl4ai(t: dict[str, str]) -> str:
    w, h = 980, 430
    body = text(24, 38, "jevcrawl vs Crawl4AI", t["text"], 22, 700)
    body += text(24, 62, "20 WebWalkerQA tasks · both crawl toward the question without an LLM · "
                 "30-page budget", t["muted"], 14)  # fmt: skip
    x0, bar_max = 24, 330
    for m, (metric, jev, c4, top_v, unit, lower) in enumerate(VS_CRAWL4AI):
        y = 100 + m * 78
        label = metric + (" (lower is better)" if lower else "")
        body += text(x0, y, label, t["text"], 14, 600)
        for k, (name, v, fill) in enumerate(
            [("jevcrawl", jev, "url(#g)"), ("Crawl4AI", c4, t["other2"])]
        ):
            by = y + 10 + k * 22
            body += text(x0, by + 13, name, t["muted"], 13)
            bw = bar_max * v / top_v
            body += bar(x0 + 80, by + 2, bw, 14, fill)
            better = (v <= (c4 if k == 0 else jev)) if lower else (v >= (c4 if k == 0 else jev))
            body += text(x0 + 80 + max(bw, 3) + 10, by + 14, f"{v}{unit}", t["text"], 14,
                         800 if better else 400)  # fmt: skip
    # Which tasks each one reached: one square per task.
    px = 560
    body += text(px, 100, "Who found the answer page, task by task", t["text"], 14, 600)
    kinds = [
        ("only_jev", "url(#g)", None, "Only jevcrawl"),
        ("both", ACCENT_A, ".45", "Both"),
        ("only_c4", t["other2"], None, "Only Crawl4AI"),
        ("neither", t["empty"], None, "Neither"),
    ]
    cells = [(fill, op) for key, fill, op, _ in kinds for _ in range(OVERLAP[key])]
    for i, (fill, op) in enumerate(cells):
        cx, cy = px + (i % 10) * 38, 116 + (i // 10) * 38
        opacity = f' fill-opacity="{op}"' if op else ""
        body += rect(cx, cy, 32, 32, 7, fill, opacity)
    for j, (key, fill, op, label) in enumerate(kinds):
        lx, ly = px + (j % 2) * 190, 216 + (j // 2) * 30
        opacity = f' fill-opacity="{op}"' if op else ""
        body += rect(lx, ly - 13, 16, 16, 4, fill, opacity)
        body += text(lx + 24, ly, f"{label}: {OVERLAP[key]}", t["text"], 14)
    body += text(px, 296, "jevcrawl found 6 answer pages Crawl4AI missed;", t["text"], 14, 600)
    body += text(px, 318, "Crawl4AI found 1 that jevcrawl missed.", t["text"], 14, 600)
    body += text(24, h - 18, "Crawl4AI 0.9.4, BestFirstCrawlingStrategy + KeywordRelevanceScorer, "
                 "depth 3, nav/footer/aside removed. Its 3 failures were sites its browser "
                 "could not load.", t["muted"], 12)  # fmt: skip
    return svg(w, h, body, "jevcrawl vs Crawl4AI")


STEPS = [
    # title, description, uses Jev
    ("Seed", "Sitemap URLs that share words with the goal", False),
    ("Fetch", "The best 4 pages at a time, HTML only, 5 MB cap", False),
    ("Segment", "Passages with heading path and character offsets", False),
    ("Filter for free", "Boilerplate, link lists, login and cart pages, other hosts", False),
    ("Score", "One request per page: every passage, every link", True),
    ("Stop?", "Do the kept passages answer the goal?", True),
]


def wrap(s: str, width: int) -> list[str]:
    lines: list[str] = []
    for word in s.split():
        if lines and len(lines[-1]) + 1 + len(word) <= width:
            lines[-1] += " " + word
        else:
            lines.append(word)
    return lines


def how_it_works(t: dict[str, str]) -> str:
    w, h, top, card_w, card_h, gap = 980, 410, 92, 144, 164, 15.2
    body = text(24, 38, "How a crawl works", t["text"], 22, 700)
    body += text(24, 62, "No LLM in the loop: Jev scores, plain code decides what to fetch "
                 "and what to keep", t["muted"], 14)  # fmt: skip
    centers = []
    for i, (title, desc, jev) in enumerate(STEPS):
        x = 20 + i * (card_w + gap)
        centers.append(x + card_w / 2)
        if jev:
            body += rect(x, top, card_w, card_h, 14, t["row"], ' stroke="url(#g)" stroke-width="2"')
            body += rect(x + card_w - 50, top + 14, 38, 20, 10, "url(#g)")
            body += text(x + card_w - 31, top + 28, "Jev", "#fff", 12, 700, "middle")
        else:
            body += rect(x, top, card_w, card_h, 14, "none", f' stroke="{t["grid"]}"')
        fill = "url(#g)" if jev else t["other2"]
        body += f'<circle cx="{x + 26}" cy="{top + 24}" r="13" fill="{fill}"/>\n'
        body += text(x + 26, top + 29, str(i + 1), "#fff", 13, 700, "middle")
        body += text(x + 14, top + 66, title, t["text"], 16, 700)
        for k, line in enumerate(wrap(desc, 19)):
            body += text(x + 14, top + 92 + k * 18, line, t["muted"], 13)
        if i < len(STEPS) - 1:  # arrow to the next step
            ax, ay = x + card_w + 3, top + card_h / 2
            body += (
                f'<path d="M{ax:.1f} {ay - 6} l7 6 l-7 6" fill="none" stroke="{t["other2"]}" '
                'stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"/>\n'
            )
    # Not answered yet: back to fetching.
    y0, y1 = top + card_h, top + card_h + 34
    body += (
        f'<path d="M{centers[5]:.1f} {y0} V{y1} H{centers[1]:.1f} V{y0 + 8}" fill="none" '
        f'stroke="url(#g)" stroke-width="2.5" stroke-dasharray="6 5" stroke-linecap="round"/>\n'
        f'<path d="M{centers[1] - 7:.1f} {y0 + 14} l7 -8 l7 8" fill="none" stroke="{ACCENT_A}" '
        'stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"/>\n'
    )
    mid = (centers[1] + centers[5]) / 2
    body += rect(mid - 140, y1 - 13, 280, 26, 13, t["bg"])
    body += text(mid, y1 + 5, "not answered yet: follow the next best links", t["text"], 13, 600,
                 "middle")  # fmt: skip
    # What the agent gets.
    oy = 326
    body += rect(20, oy, w - 40, 64, 14, t["row"])
    body += text(40, oy + 27, "Your agent gets", t["text"], 15, 700)
    body += text(40, oy + 48, "Verbatim passages, source URL, heading path, character offsets",
                 t["muted"], 13)  # fmt: skip
    body += (
        f'<text x="{w - 40}" y="{oy + 30}" font-size="22" font-weight="800" text-anchor="end">'
        f'<tspan fill="{t["muted"]}">80,034</tspan><tspan fill="{t["text"]}"> → </tspan>'
        f'<tspan fill="url(#g)">1,873 tokens</tspan></text>\n'
    )
    body += text(w - 40, oy + 50, "docs.python.org example · $0.0069 · 8.3 s", t["muted"], 13,
                 400, "end")  # fmt: skip
    return svg(w, h, body, "How a jevcrawl crawl works")


# The real docs.python.org crawl from the README (2026-10-12), in the CLI's output format.
DEMO_COMMAND = [
    "jevcrawl crawl https://docs.python.org/3/ \\",
    '    --goal "What does the walrus operator := do and which Python version added it?"',
]
DEMO_OUTPUT = [
    # text, colour key
    ("[kept 0.97] https://docs.python.org/3/whatsnew/3.8.html [307:580]", "accent"),
    ("  (What\u2019s New In Python 3.8 > New Features > Assignment expressions)", "dim"),
    ("  There is new syntax := that assigns values to variables as part of a larger", "fg"),
    ("  expression. It is affectionately known as “the walrus operator” due to its", "fg"),
    ("  resemblance to the eyes and tusks of a walrus.", "fg"),
    ("  ⋮", "dim"),
    ("", "fg"),
    ("pages=6 stopped=answered tokens=80,034->1,873 jev_calls=11 cost=$0.0069 time=8.3s", "ok"),
]
MONO = "ui-monospace, SFMono-Regular, Menlo, Consolas, 'Liberation Mono', monospace"


def _cycle(values: list[str], times: list[float], period: float, attr: str = "opacity") -> str:
    key_times = ";".join(f"{t / period:.4f}" for t in times)
    return (
        f'<animate attributeName="{attr}" values="{";".join(values)}" keyTimes="{key_times}" '
        f'dur="{period}s" repeatCount="indefinite" calcMode="discrete"/>'
    )


def demo() -> str:
    """A looping terminal: the command is typed, then the result appears."""
    from html import escape

    w, h, period, x, lh = 980, 400, 16.0, 28, 23
    colors = {"fg": "#E6EDF3", "dim": "#7D8590", "accent": "#A5A0FF", "ok": "#3FD0F0"}
    body = rect(0, 0, w, h, 14, "#0D1117", ' stroke="#30363D"')
    for i, dot in enumerate(["#FF5F57", "#FEBC2E", "#28C840"]):
        body += f'<circle cx="{24 + i * 20}" cy="22" r="6" fill="{dot}"/>\n'
    body += (
        f'<text x="{w / 2}" y="27" fill="#7D8590" font-size="13" text-anchor="middle">'
        "jevcrawl · replay of a real crawl</text>\n"
    )
    body += f"<style>.m text{{font-family:{MONO};font-size:14px;white-space:pre}}</style>\n"
    body += '<g class="m">\n'
    y, start, char_s = 70.0, 0.6, 0.028
    for i, line in enumerate(DEMO_COMMAND):  # typed one character at a time
        shown = ("$ " if i == 0 else "") + line
        prefix = 2 if i == 0 else 0
        steps = list(range(prefix, len(shown) + 1))
        times = [0.0] + [start + k * char_s for k in range(len(steps))] + [period - 0.4]
        clip = f"c{i}"  # a clip that widens one character at a time
        cw = 8.45  # monospace advance at 14px; the last frame reveals the whole line anyway
        widths = ["0"] + [f"{n * cw:.1f}" for n in steps[:-1]] + [str(w)] + ["0"]
        body += (
            f'<clipPath id="{clip}"><rect x="{x}" y="{y - 16}" height="22" width="0">'
            f"{_cycle(widths, times, period, 'width')}</rect></clipPath>\n"
            f'<text x="{x}" y="{y}" fill="{colors["fg"]}" clip-path="url(#{clip})">'
            f"{escape(shown)}</text>\n"
        )
        start = times[-2] + 0.25
        y += lh
    y += lh / 2
    t = start + 1.2  # the crawl itself
    bold = ' font-weight="700"'
    for line, color in DEMO_OUTPUT:
        times = [0.0, t, period - 0.4]
        body += (
            f'<text x="{x}" y="{y}" fill="{colors[color]}" opacity="0"'
            f"{bold if color == 'ok' else ''}>{escape(line)}"
            f"{_cycle(['0', '1', '0'], times, period)}</text>\n"
        )
        t += 0.12
        y += lh
    prompt_t = t + 0.3
    body += (
        f'<text x="{x}" y="{y + 8}" fill="{colors["fg"]}" opacity="0">$ '
        f"{_cycle(['0', '1', '0'], [0.0, prompt_t, period - 0.4], period)}</text>\n"
    )
    blink = [0.0, prompt_t] + [prompt_t + 0.5 * k for k in range(1, 30) if prompt_t + 0.5 * k
                               < period - 0.4] + [period - 0.4]  # fmt: skip
    states = ["0"] + ["1" if k % 2 == 0 else "0" for k in range(len(blink) - 2)] + ["0"]
    body += (
        f'<rect x="{x + 18}" y="{y - 6}" width="9" height="17" fill="{ACCENT_B}" opacity="0">'
        f"{_cycle(states, blink, period)}</rect>\n</g>\n"
    )
    # The point of it, once the result is in.
    callout = (
        f'<g opacity="0">{_cycle(["0", "1", "0"], [0.0, t + 0.6, period - 0.4], period)}'
        + rect(w - 372, h - 70, 344, 46, 23, "url(#g)")
        + text(w - 200, h - 41, "80,034 → 1,873 tokens for $0.0069", "#fff", 16, 700, "middle")
        + "</g>\n"
    )
    return svg(w, h, body + callout, "jevcrawl demo: a real crawl of docs.python.org")


def social() -> str:
    """GitHub's social preview card (Settings > Social preview), 1280x640."""
    w, h = 1280, 640
    body = rect(0, 0, w, h, 0, "#0D1117")
    body += '<circle cx="1180" cy="40" r="360" fill="url(#g)" opacity=".16"/>\n'
    body += mark(96, 84, 150)
    body += (
        '<text x="276" y="194" font-size="104" font-weight="800" letter-spacing="-3">'
        '<tspan fill="url(#g)">jev</tspan><tspan fill="#F0F6FC">crawl</tspan></text>\n'
    )
    body += text(96, 312, "The web crawler for AI agents that finds", "#F0F6FC", 44, 700)
    body += text(96, 366, "the answer, not just pages.", "#F0F6FC", 44, 700)
    stats = [
        ("2x", ["answer pages found", "vs Crawl4AI"]),
        ("10x", ["cheaper per answer", "than Claude crawling"]),
        ("MCP", ["Claude Code, Cursor,", "Claude Desktop"]),
    ]
    for i, (big, lines) in enumerate(stats):
        x = 96 + i * 370
        body += text(x, 462, big, ACCENT_B if i != 1 else "#A5A0FF", 56, 800)
        for k, line in enumerate(lines):
            body += text(x, 500 + k * 28, line, "#9198A1", 22)
    body += text(96, 598, "github.com/RayenMlayeh/jevcrawl · Apache-2.0", "#6E7781", 20)
    return svg(w, h, body, "jevcrawl")


def main() -> None:
    for name, t in THEMES.items():
        for stem, draw in [("logo", logo), ("scoreboard", scoreboard), ("cost", cost_per_answer),
                           ("crawl4ai", crawl4ai), ("how", how_it_works)]:  # fmt: skip
            (OUT / f"{stem}-{name}.svg").write_text(draw(t), encoding="utf-8")
    (OUT / "mark.svg").write_text(svg(120, 120, mark(0, 0, 120), "jevcrawl"), encoding="utf-8")
    (OUT / "demo.svg").write_text(demo(), encoding="utf-8")
    (OUT / "social.svg").write_text(social(), encoding="utf-8")


if __name__ == "__main__":
    main()
Source README
jevcrawl

The web crawler for AI agents that finds the answer, not just pages.

A small decision model picks every link and every passage, so your agent's LLM reads
only what answers its question at a few cents per crawl.

PyPI
CI
License
Python
MCP server
Benchmark
Cost

Quick start · Benchmarks · Tools ·
How it works · Configuration


Demo: jevcrawl crawls docs.python.org for 'What does the walrus operator := do and which Python version added it?' and returns the What's New In Python 3.8 passage. 6 pages, 80,034 tokens reduced to 1,873, $0.0069, 8.3 seconds.

On 50 WebWalkerQA tasks jevcrawl answered 40% correctly and found the answer page 54% of the time for $0.025 per task, against 30%, 33% and $0.194 when Claude picks every link and 8%, 11% and $0.228 for a breadth-first crawl.

Why jevcrawl

Agents that browse a site have two bad options. They can read everything a crawler fetches,
which buries the answer in navigation and boilerplate and burns tokens. Or they can let the LLM
pick each link
, which is slow and costs dollars per question. jevcrawl does neither:

  • 🎯 Finds the answer page more often. 54% of tasks vs 33% when Claude picks the links, and
    2x as often as Crawl4AI's goal-directed crawl.
  • 💸 About 8x cheaper per task, 10x per correct answer. The crawl decisions cost about a cent.
  • ⚡ 2x faster than an LLM-driven crawl. Four pages at a time, one decision request per page.
  • 📜 Verbatim, with sources. Every passage is copied from the page with its URL, heading path
    and character offsets. The decision model only scores, so it cannot invent text.
  • 🛡️ Safe against prompt injection by design. Page text is only ever scored, never followed.
  • 🔌 Works everywhere MCP does: Claude Code, Claude Desktop, Cursor, or the CLI.

Quick start

You need a Jev API key from console.typesafe.ai and
uv.

Claude Code

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

Claude Desktop, Cursor and other MCP clients (mcpServers in the client's config file)

{
  "mcpServers": {
    "jevcrawl": {
      "command": "uvx",
      "args": ["jevcrawl"],
      "env": { "TYPESAFE_API_KEY": "<your key>" }
    }
  }
}

Then ask your agent something like "Use jevcrawl to find which Python version added the walrus
operator, starting from docs.python.org"
. Check the setup with
uvx jevcrawl doctor. Agents that load skills
can also use skills/jevcrawl/SKILL.md.

Pages that need JavaScript (optional). jevcrawl reads plain HTML. To also render pages that
are empty without JavaScript, add the js extra and download Chromium once:

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx --from "jevcrawl[js]" jevcrawl
uvx --from "jevcrawl[js]" playwright install chromium

(In mcpServers, use "args": ["--from", "jevcrawl[js]", "jevcrawl"].) Only pages detected as
JavaScript shells are opened in the browser, at most five per call, and they are listed in
stats.js_rendered_urls. Everything else is still fetched as plain HTML.

Benchmarks

Every system starts on a site's home page with the same 30-page budget and the same question.
What each one collects goes to the same answering LLM, and the same judge grades the
answer, so the crawler is the only thing that changes.

Cost of one correct answer: jevcrawl $0.06, Claude picking every link $0.65, breadth-first crawl $2.85.

jevcrawl vs Crawl4AI

Crawl4AI is one of the most popular open-source crawlers
for LLMs. Its BestFirstCrawlingStrategy with a keyword scorer is its way to crawl toward a question
without an LLM, so we compared the two on whether they fetch the page that holds the answer.

On 20 tasks jevcrawl found an answer page 10 times and Crawl4AI 5 times; every answer page 5 vs 2; median crawl time 29 s vs 43 s; sites that failed to load 0 vs 3. jevcrawl found 6 answer pages Crawl4AI missed, Crawl4AI found 1 jevcrawl missed.

Where the numbers come from

Tasks WebWalkerQA from WebWalker: Benchmarking LLMs in Web Traversal (Wu et al., Alibaba Tongyi Lab, ACL 2025): questions whose answers sit several clicks deep in real websites (conferences, universities, organizations, games), in English and Chinese.
Task selection Each task was re-checked live before the run: the site and answer page still load and still contain the reference answer. Tasks looked at while developing are excluded (bench/tasks/seen-ids.txt).
Accuracy run 50 tasks, 2026-10-12. Answering model claude-opus-5-5, judge claude-sonnet-5-5.
Crawl4AI run 20 different tasks, 2026-10-13, Crawl4AI 0.9.4. Crawl only, no LLM: did the crawler fetch the gold page? In the accuracy run, whether a system fetched the gold page closely tracked whether it answered correctly.
Baselines Claude picks every link: Claude reads each page and chooses the next link. Breadth-first: every page to depth 2. All systems share jevcrawl's fetcher and remove navigation and footers.
Costs Jev cost is billed and exact. LLM costs are token counts at list price.
Caveats 50 and 20 tasks are small samples: the cost gap is large and stable, the accuracy gap less so. Crawl4AI drives a real browser; the benchmark ran jevcrawl without JavaScript rendering.

The benchmark code and how to reproduce every number: bench/.

Tools

Tool Use it for
crawl(goal, start_url, max_pages=30, max_seconds=120, return_tokens=4000, max_decision_usd=0.05) The answer may be several pages deep. Returns the relevant passages.
scrape(url, goal=None) One known page. With a goal, only the relevant passages; without one, all content and no Jev call.
map(start_url, goal, limit=50, max_pages=10) Ranked URLs only, to decide what to scrape.
A real result (docs.python.org, "What does the walrus operator := do and which Python version added it?")
{"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
             "heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
             "text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
             "char_start": 307, "char_end": 580, "relevance": 0.97}],
 "uncertain": [],
 "pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
            "text_sha256": "b624ce93..."}],
 "stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
           "cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
           "model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
           "needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}

Trimmed to one block and one page. 80,034 tokens of pages became 1,873 tokens of answer, for
$0.0069, in 8.3 seconds.

  • blocks are the relevant passages, best first; uncertain are borderline ones worth reading
    before concluding something is missing.
  • char_start/char_end index into the page text whose sha256 is in pages, so a quote can be
    checked later.
  • stopped_because is answered, frontier_empty, max_pages, max_seconds, budget or
    auth_error.
  • Expected failures come back as {"error": "<code>", "message": "<what to do>"}; the tools
    never raise into the agent's session.

The same operations work from the command line (uv tool install jevcrawl or
pip install jevcrawl puts jevcrawl on your path):

jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
jevcrawl map https://example.com --goal "..." [--limit 20]
jevcrawl doctor

How it works

A crawl in six steps: seed from the sitemap, fetch the best 4 pages, segment them into passages, filter boilerplate for free, Jev scores every passage and link in one request per page, Jev checks whether the kept passages answer the goal. If not, it follows the next best links. The docs.python.org example turned 80,034 tokens of pages into 1,873 tokens for $0.0069 in 8.3 s.
  • Jev decides, it does not write. Jev returns a probability per
    yes/no question, so it cannot produce text. Everything you get back is copied from the page.
  • Page text is data, never instructions. It travels only in data fields of a question; every
    question sentence is a fixed string written by jevcrawl. An injected "ignore previous
    instructions" can at worst bias a score.
  • Links below the priority floor are kept in reserve, not dropped, so a crawl never stops
    early while it still has page budget.
  • Failures degrade, they do not hide. After three failed Jev requests the crawl continues
    breadth-first, returns every block unscored in uncertain, and sets stats.degraded.
  • Repeats are free. Jev's answers are cached locally (~/.jevcrawl/cache), so the same goal
    on the same pages costs nothing.

Configuration

Environment variable Meaning Default
TYPESAFE_API_KEY Jev API key (required) none
JEVCRAWL_BASE_URL Jev-compatible endpoint https://api.typesafe.ai
JEVCRAWL_MODEL model id jev-1.13.0 (pinned: thresholds are calibrated per version)
JEVCRAWL_CACHE_DIR answer cache ~/.jevcrawl/cache
JEVCRAWL_RESPECT_ROBOTS skip URLs the site's robots.txt disallows (0 turns it off) 1
JEVCRAWL_RENDER_JS render JavaScript-only pages when the js extra is installed (0 turns it off) 1

Everything else (thresholds, links per page, concurrency) has a measured default in
jevcrawl/config.py, and Python callers can pass their own Config.

Cost and privacy

Jev charges for input tokens only, $0.042 per million for jev-1.13.0; in the benchmark a crawl
cost $0.011 on average. max_decision_usd is a hard limit: a request that could pass it is never
sent, and the crawl stops with budget. Every result reports jev_cost_usd, and tokens_raw
(what a plain crawl would hand your LLM) against tokens_returned.

Page text is sent to TypeSafe (blocks, link labels and URLs) to be scored. The API key is
read from the environment and never written to disk or logs.

Crawling responsibly

jevcrawl follows each site's robots.txt by default (set JEVCRAWL_RESPECT_ROBOTS=0 to turn
that off) and does not rate-limit itself beyond fetching four pages at a time. You are
responsible for the sites you crawl, their terms of service and the law that applies to you. It
never logs in and does not try to get past paywalls or CAPTCHAs. Without the js extra it does
not run JavaScript (pages that need it are listed in stats.needs_js_urls). Every call is
bounded by max_pages, max_seconds, max_decision_usd and return_tokens.

assets/build.py

"""Draw the README logo and benchmark charts as SVG, in a light and a dark version.

Every number below was measured; where each comes from is in bench/README.md. After a new
benchmark run, update the numbers here and run ``uv run python assets/build.py``.
"""

from __future__ import annotations

from pathlib import Path

OUT = Path(__file__).resolve().parent
FONT = "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif"

THEMES = {
    "light": {"text": "#1F2328", "muted": "#59636E", "grid": "#D1D9E0", "other": "#AEB8C2",
              "other2": "#7D8894", "row": "#6D5DFC14", "empty": "#EEF1F4", "bg": "#FFFFFF"},
    "dark": {"text": "#F0F6FC", "muted": "#9198A1", "grid": "#3D444D", "other": "#4A535D",
             "other2": "#6E7781", "row": "#8B7FFF1F", "empty": "#262C33", "bg": "#0D1117"},
}  # fmt: skip
ACCENT_A, ACCENT_B = "#6D5DFC", "#22C3EE"

# 50 live-verified WebWalkerQA tasks, same answering LLM and judge for every system (2026-10-12).
SCOREBOARD = [
    # name, accuracy %, found answer page %, cost per task $, seconds per task
    ("jevcrawl", 40, 54, 0.025, 27.5),
    ("Claude picks every link", 30, 33, 0.194, 61.7),
    ("Breadth-first crawl", 8, 11, 0.228, 22.4),
]
# 20 other verified tasks, crawl only, no LLM (bench/reach.py, 2026-10-13).
VS_CRAWL4AI = [
    # metric, jevcrawl, Crawl4AI, scale max, unit, lower is better
    ("Found a page with the answer", 10, 5, 20, "/20", False),
    ("Found every page the answer needs", 5, 2, 20, "/20", False),
    ("Median crawl time", 29, 43, 43, " s", True),
    ("Sites it could not load", 0, 3, 3, "/20", True),
]
OVERLAP = {"only_jev": 6, "both": 4, "only_c4": 1, "neither": 9}


def svg(width: int, height: int, body: str, title: str) -> str:
    return (
        f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
        f'viewBox="0 0 {width} {height}" role="img" aria-label="{title}">\n'
        f"<title>{title}</title>\n"
        f"<style>text{{font-family:{FONT}}}</style>\n"
        f'<defs><linearGradient id="g" x1="0" y1="0" x2="1" y2="1">'
        f'<stop offset="0" stop-color="{ACCENT_A}"/><stop offset="1" stop-color="{ACCENT_B}"/>'
        f"</linearGradient></defs>\n{body}</svg>\n"
    )


def text(x: float, y: float, s: str, fill: str, size: int = 14, weight: int = 400,
         anchor: str = "start") -> str:  # fmt: skip
    return (
        f'<text x="{x:.1f}" y="{y:.1f}" fill="{fill}" font-size="{size}" '
        f'font-weight="{weight}" text-anchor="{anchor}">{s}</text>\n'
    )


def rect(x: float, y: float, w: float, h: float, rx: float, fill: str, extra: str = "") -> str:
    return (
        f'<rect x="{x:.1f}" y="{y:.1f}" width="{w:.1f}" height="{h}" rx="{rx}" '
        f'fill="{fill}"{extra}/>\n'
    )


def bar(x: float, y: float, w: float, h: float, fill: str) -> str:
    return rect(x, y, max(w, 3), h, h / 2, fill)


def mark(x: float, y: float, size: float) -> str:
    """A crawl graph: many branches, one lit path to the answer."""
    s = size / 120
    faint = [((26, 92), (30, 58)), ((26, 92), (58, 98)), ((48, 70), (76, 82)), ((70, 48), (60, 22))]
    lit = [(26, 92), (48, 70), (70, 48), (94, 26)]
    p = f'<g transform="translate({x},{y}) scale({s:.4f})">\n'
    p += '<rect width="120" height="120" rx="28" fill="url(#g)"/>\n'
    for (x1, y1), (x2, y2) in faint:
        p += (
            f'<line x1="{x1}" y1="{y1}" x2="{x2}" y2="{y2}" stroke="#fff" stroke-opacity=".38" '
            'stroke-width="4" stroke-linecap="round"/>\n'
            f'<circle cx="{x2}" cy="{y2}" r="5" fill="#fff" fill-opacity=".45"/>\n'
        )
    points = " ".join(f"{a},{b}" for a, b in lit)
    p += (
        f'<polyline points="{points}" fill="none" stroke="#fff" stroke-width="7" '
        'stroke-linecap="round" stroke-linejoin="round"/>\n'
    )
    for a, b in lit[:-1]:
        p += f'<circle cx="{a}" cy="{b}" r="7" fill="#fff"/>\n'
    p += '<circle cx="94" cy="26" r="15" fill="none" stroke="#fff" stroke-width="4"/>\n'
    p += '<circle cx="94" cy="26" r="8" fill="#fff"/>\n</g>\n'
    return p


def logo(t: dict[str, str]) -> str:
    body = mark(8, 8, 104)
    body += (
        f'<text x="134" y="84" font-size="72" font-weight="800" letter-spacing="-2">'
        f'<tspan fill="url(#g)">jev</tspan><tspan fill="{t["text"]}">crawl</tspan></text>\n'
    )
    return svg(470, 120, body, "jevcrawl")


def scoreboard(t: dict[str, str]) -> str:
    w, row_h, top = 980, 58, 112
    h = top + row_h * len(SCOREBOARD) + 44
    cols = [
        ("Answered correctly", 1, 100.0, lambda v: f"{v}%", False),
        ("Found the answer page", 2, 100.0, lambda v: f"{v}%", False),
        ("Cost per task", 3, 0.25, lambda v: f"${v:.3f}", True),
        ("Time per task", 4, 70.0, lambda v: f"{v:.1f} s", True),
    ]
    body = text(24, 38, "50 real websites, one question each", t["text"], 22, 700)
    body += text(24, 62, "WebWalkerQA tasks verified live · same answering LLM and judge for "
                 "every system · 30-page budget", t["muted"], 14)  # fmt: skip
    x0, col_w, bar_max = 236, 186, 104
    for i, (name, _, _, _, lower) in enumerate(cols):
        cx = x0 + i * col_w
        body += text(cx, top - 16, name, t["text"], 14, 600)
        if lower:
            body += text(cx + col_w - 18, top - 16, "lower is better", t["muted"], 11, 400, "end")
    for r, row in enumerate(SCOREBOARD):
        y = top + r * row_h
        mine = r == 0
        if mine:
            body += rect(12, y, w - 24, row_h - 6, 12, t["row"])
        body += text(24, y + 32, row[0], t["text"], 16, 700 if mine else 500)
        for i, (_, k, top_v, fmt, lower) in enumerate(cols):
            values = [s[k] for s in SCOREBOARD]
            best = (min if lower else max)(values)
            v = row[k]
            cx = x0 + i * col_w
            fill = "url(#g)" if mine else t["other"]
            body += bar(cx, y + 18, bar_max * v / top_v, 16, fill)
            body += text(cx + bar_max * v / top_v + 10, y + 31, fmt(v),
                         t["text"], 15, 800 if v == best else 400)  # fmt: skip
    body += text(24, h - 18, "Bold: best in column. Cost per task includes the answering LLM's "
                 "tokens at list price; jevcrawl's own Jev calls were $0.011.",
                 t["muted"], 12)  # fmt: skip
    return svg(w, h, body, "jevcrawl benchmark scoreboard")


def cost_per_answer(t: dict[str, str]) -> str:
    rows = [(name, cost / (acc / 100)) for name, acc, _, cost, _ in SCOREBOARD]
    w, h, x0, bar_max = 980, 320, 236, 560
    body = text(24, 38, "What one correct answer costs", t["text"], 22, 700)
    body += text(24, 62, "Cost per task ÷ accuracy, same 50 tasks", t["muted"], 14)
    top_v = max(c for _, c in rows)
    for r, (name, c) in enumerate(rows):
        y = 96 + r * 58
        mine = r == 0
        if mine:
            body += rect(12, y - 4, w - 24, 50, 12, t["row"])
        body += text(24, y + 26, name, t["text"], 16, 700 if mine else 500)
        bw = bar_max * c / top_v
        body += bar(x0, y + 10, bw, 20, "url(#g)" if mine else t["other"])
        body += text(x0 + max(bw, 3) + 12, y + 26, f"${c:.2f}", t["text"], 16, 800 if mine else 400)
    jev, claude, bfs = (c for _, c in rows)
    note = (
        f"jevcrawl: {claude / jev:.0f}x less than letting Claude pick the links, "
        f"{bfs / jev:.0f}x less than a breadth-first crawl"
    )
    body += text(24, h - 24, note, t["text"], 15, 600)
    return svg(w, h, body, "Cost per correct answer")


def crawl4ai(t: dict[str, str]) -> str:
    w, h = 980, 430
    body = text(24, 38, "jevcrawl vs Crawl4AI", t["text"], 22, 700)
    body += text(24, 62, "20 WebWalkerQA tasks · both crawl toward the question without an LLM · "
                 "30-page budget", t["muted"], 14)  # fmt: skip
    x0, bar_max = 24, 330
    for m, (metric, jev, c4, top_v, unit, lower) in enumerate(VS_CRAWL4AI):
        y = 100 + m * 78
        label = metric + (" (lower is better)" if lower else "")
        body += text(x0, y, label, t["text"], 14, 600)
        for k, (name, v, fill) in enumerate(
            [("jevcrawl", jev, "url(#g)"), ("Crawl4AI", c4, t["other2"])]
        ):
            by = y + 10 + k * 22
            body += text(x0, by + 13, name, t["muted"], 13)
            bw = bar_max * v / top_v
            body += bar(x0 + 80, by + 2, bw, 14, fill)
            better = (v <= (c4 if k == 0 else jev)) if lower else (v >= (c4 if k == 0 else jev))
            body += text(x0 + 80 + max(bw, 3) + 10, by + 14, f"{v}{unit}", t["text"], 14,
                         800 if better else 400)  # fmt: skip
    # Which tasks each one reached: one square per task.
    px = 560
    body += text(px, 100, "Who found the answer page, task by task", t["text"], 14, 600)
    kinds = [
        ("only_jev", "url(#g)", None, "Only jevcrawl"),
        ("both", ACCENT_A, ".45", "Both"),
        ("only_c4", t["other2"], None, "Only Crawl4AI"),
        ("neither", t["empty"], None, "Neither"),
    ]
    cells = [(fill, op) for key, fill, op, _ in kinds for _ in range(OVERLAP[key])]
    for i, (fill, op) in enumerate(cells):
        cx, cy = px + (i % 10) * 38, 116 + (i // 10) * 38
        opacity = f' fill-opacity="{op}"' if op else ""
        body += rect(cx, cy, 32, 32, 7, fill, opacity)
    for j, (key, fill, op, label) in enumerate(kinds):
        lx, ly = px + (j % 2) * 190, 216 + (j // 2) * 30
        opacity = f' fill-opacity="{op}"' if op else ""
        body += rect(lx, ly - 13, 16, 16, 4, fill, opacity)
        body += text(lx + 24, ly, f"{label}: {OVERLAP[key]}", t["text"], 14)
    body += text(px, 296, "jevcrawl found 6 answer pages Crawl4AI missed;", t["text"], 14, 600)
    body += text(px, 318, "Crawl4AI found 1 that jevcrawl missed.", t["text"], 14, 600)
    body += text(24, h - 18, "Crawl4AI 0.9.4, BestFirstCrawlingStrategy + KeywordRelevanceScorer, "
                 "depth 3, nav/footer/aside removed. Its 3 failures were sites its browser "
                 "could not load.", t["muted"], 12)  # fmt: skip
    return svg(w, h, body, "jevcrawl vs Crawl4AI")


STEPS = [
    # title, description, uses Jev
    ("Seed", "Sitemap URLs that share words with the goal", False),
    ("Fetch", "The best 4 pages at a time, HTML only, 5 MB cap", False),
    ("Segment", "Passages with heading path and character offsets", False),
    ("Filter for free", "Boilerplate, link lists, login and cart pages, other hosts", False),
    ("Score", "One request per page: every passage, every link", True),
    ("Stop?", "Do the kept passages answer the goal?", True),
]


def wrap(s: str, width: int) -> list[str]:
    lines: list[str] = []
    for word in s.split():
        if lines and len(lines[-1]) + 1 + len(word) <= width:
            lines[-1] += " " + word
        else:
            lines.append(word)
    return lines


def how_it_works(t: dict[str, str]) -> str:
    w, h, top, card_w, card_h, gap = 980, 410, 92, 144, 164, 15.2
    body = text(24, 38, "How a crawl works", t["text"], 22, 700)
    body += text(24, 62, "No LLM in the loop: Jev scores, plain code decides what to fetch "
                 "and what to keep", t["muted"], 14)  # fmt: skip
    centers = []
    for i, (title, desc, jev) in enumerate(STEPS):
        x = 20 + i * (card_w + gap)
        centers.append(x + card_w / 2)
        if jev:
            body += rect(x, top, card_w, card_h, 14, t["row"], ' stroke="url(#g)" stroke-width="2"')
            body += rect(x + card_w - 50, top + 14, 38, 20, 10, "url(#g)")
            body += text(x + card_w - 31, top + 28, "Jev", "#fff", 12, 700, "middle")
        else:
            body += rect(x, top, card_w, card_h, 14, "none", f' stroke="{t["grid"]}"')
        fill = "url(#g)" if jev else t["other2"]
        body += f'<circle cx="{x + 26}" cy="{top + 24}" r="13" fill="{fill}"/>\n'
        body += text(x + 26, top + 29, str(i + 1), "#fff", 13, 700, "middle")
        body += text(x + 14, top + 66, title, t["text"], 16, 700)
        for k, line in enumerate(wrap(desc, 19)):
            body += text(x + 14, top + 92 + k * 18, line, t["muted"], 13)
        if i < len(STEPS) - 1:  # arrow to the next step
            ax, ay = x + card_w + 3, top + card_h / 2
            body += (
                f'<path d="M{ax:.1f} {ay - 6} l7 6 l-7 6" fill="none" stroke="{t["other2"]}" '
                'stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"/>\n'
            )
    # Not answered yet: back to fetching.
    y0, y1 = top + card_h, top + card_h + 34
    body += (
        f'<path d="M{centers[5]:.1f} {y0} V{y1} H{centers[1]:.1f} V{y0 + 8}" fill="none" '
        f'stroke="url(#g)" stroke-width="2.5" stroke-dasharray="6 5" stroke-linecap="round"/>\n'
        f'<path d="M{centers[1] - 7:.1f} {y0 + 14} l7 -8 l7 8" fill="none" stroke="{ACCENT_A}" '
        'stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"/>\n'
    )
    mid = (centers[1] + centers[5]) / 2
    body += rect(mid - 140, y1 - 13, 280, 26, 13, t["bg"])
    body += text(mid, y1 + 5, "not answered yet: follow the next best links", t["text"], 13, 600,
                 "middle")  # fmt: skip
    # What the agent gets.
    oy = 326
    body += rect(20, oy, w - 40, 64, 14, t["row"])
    body += text(40, oy + 27, "Your agent gets", t["text"], 15, 700)
    body += text(40, oy + 48, "Verbatim passages, source URL, heading path, character offsets",
                 t["muted"], 13)  # fmt: skip
    body += (
        f'<text x="{w - 40}" y="{oy + 30}" font-size="22" font-weight="800" text-anchor="end">'
        f'<tspan fill="{t["muted"]}">80,034</tspan><tspan fill="{t["text"]}"> → </tspan>'
        f'<tspan fill="url(#g)">1,873 tokens</tspan></text>\n'
    )
    body += text(w - 40, oy + 50, "docs.python.org example · $0.0069 · 8.3 s", t["muted"], 13,
                 400, "end")  # fmt: skip
    return svg(w, h, body, "How a jevcrawl crawl works")


# The real docs.python.org crawl from the README (2026-10-12), in the CLI's output format.
DEMO_COMMAND = [
    "jevcrawl crawl https://docs.python.org/3/ \\",
    '    --goal "What does the walrus operator := do and which Python version added it?"',
]
DEMO_OUTPUT = [
    # text, colour key
    ("[kept 0.97] https://docs.python.org/3/whatsnew/3.8.html [307:580]", "accent"),
    ("  (What\u2019s New In Python 3.8 > New Features > Assignment expressions)", "dim"),
    ("  There is new syntax := that assigns values to variables as part of a larger", "fg"),
    ("  expression. It is affectionately known as “the walrus operator” due to its", "fg"),
    ("  resemblance to the eyes and tusks of a walrus.", "fg"),
    ("  ⋮", "dim"),
    ("", "fg"),
    ("pages=6 stopped=answered tokens=80,034->1,873 jev_calls=11 cost=$0.0069 time=8.3s", "ok"),
]
MONO = "ui-monospace, SFMono-Regular, Menlo, Consolas, 'Liberation Mono', monospace"


def _cycle(values: list[str], times: list[float], period: float, attr: str = "opacity") -> str:
    key_times = ";".join(f"{t / period:.4f}" for t in times)
    return (
        f'<animate attributeName="{attr}" values="{";".join(values)}" keyTimes="{key_times}" '
        f'dur="{period}s" repeatCount="indefinite" calcMode="discrete"/>'
    )


def demo() -> str:
    """A looping terminal: the command is typed, then the result appears."""
    from html import escape

    w, h, period, x, lh = 980, 400, 16.0, 28, 23
    colors = {"fg": "#E6EDF3", "dim": "#7D8590", "accent": "#A5A0FF", "ok": "#3FD0F0"}
    body = rect(0, 0, w, h, 14, "#0D1117", ' stroke="#30363D"')
    for i, dot in enumerate(["#FF5F57", "#FEBC2E", "#28C840"]):
        body += f'<circle cx="{24 + i * 20}" cy="22" r="6" fill="{dot}"/>\n'
    body += (
        f'<text x="{w / 2}" y="27" fill="#7D8590" font-size="13" text-anchor="middle">'
        "jevcrawl · replay of a real crawl</text>\n"
    )
    body += f"<style>.m text{{font-family:{MONO};font-size:14px;white-space:pre}}</style>\n"
    body += '<g class="m">\n'
    y, start, char_s = 70.0, 0.6, 0.028
    for i, line in enumerate(DEMO_COMMAND):  # typed one character at a time
        shown = ("$ " if i == 0 else "") + line
        prefix = 2 if i == 0 else 0
        steps = list(range(prefix, len(shown) + 1))
        times = [0.0] + [start + k * char_s for k in range(len(steps))] + [period - 0.4]
        clip = f"c{i}"  # a clip that widens one character at a time
        cw = 8.45  # monospace advance at 14px; the last frame reveals the whole line anyway
        widths = ["0"] + [f"{n * cw:.1f}" for n in steps[:-1]] + [str(w)] + ["0"]
        body += (
            f'<clipPath id="{clip}"><rect x="{x}" y="{y - 16}" height="22" width="0">'
            f"{_cycle(widths, times, period, 'width')}</rect></clipPath>\n"
            f'<text x="{x}" y="{y}" fill="{colors["fg"]}" clip-path="url(#{clip})">'
            f"{escape(shown)}</text>\n"
        )
        start = times[-2] + 0.25
        y += lh
    y += lh / 2
    t = start + 1.2  # the crawl itself
    bold = ' font-weight="700"'
    for line, color in DEMO_OUTPUT:
        times = [0.0, t, period - 0.4]
        body += (
            f'<text x="{x}" y="{y}" fill="{colors[color]}" opacity="0"'
            f"{bold if color == 'ok' else ''}>{escape(line)}"
            f"{_cycle(['0', '1', '0'], times, period)}</text>\n"
        )
        t += 0.12
        y += lh
    prompt_t = t + 0.3
    body += (
        f'<text x="{x}" y="{y + 8}" fill="{colors["fg"]}" opacity="0">$ '
        f"{_cycle(['0', '1', '0'], [0.0, prompt_t, period - 0.4], period)}</text>\n"
    )
    blink = [0.0, prompt_t] + [prompt_t + 0.5 * k for k in range(1, 30) if prompt_t + 0.5 * k
                               < period - 0.4] + [period - 0.4]  # fmt: skip
    states = ["0"] + ["1" if k % 2 == 0 else "0" for k in range(len(blink) - 2)] + ["0"]
    body += (
        f'<rect x="{x + 18}" y="{y - 6}" width="9" height="17" fill="{ACCENT_B}" opacity="0">'
        f"{_cycle(states, blink, period)}</rect>\n</g>\n"
    )
    # The point of it, once the result is in.
    callout = (
        f'<g opacity="0">{_cycle(["0", "1", "0"], [0.0, t + 0.6, period - 0.4], period)}'
        + rect(w - 372, h - 70, 344, 46, 23, "url(#g)")
        + text(w - 200, h - 41, "80,034 → 1,873 tokens for $0.0069", "#fff", 16, 700, "middle")
        + "</g>\n"
    )
    return svg(w, h, body + callout, "jevcrawl demo: a real crawl of docs.python.org")


def social() -> str:
    """GitHub's social preview card (Settings > Social preview), 1280x640."""
    w, h = 1280, 640
    body = rect(0, 0, w, h, 0, "#0D1117")
    body += '<circle cx="1180" cy="40" r="360" fill="url(#g)" opacity=".16"/>\n'
    body += mark(96, 84, 150)
    body += (
        '<text x="276" y="194" font-size="104" font-weight="800" letter-spacing="-3">'
        '<tspan fill="url(#g)">jev</tspan><tspan fill="#F0F6FC">crawl</tspan></text>\n'
    )
    body += text(96, 312, "The web crawler for AI agents that finds", "#F0F6FC", 44, 700)
    body += text(96, 366, "the answer, not just pages.", "#F0F6FC", 44, 700)
    stats = [
        ("2x", ["answer pages found", "vs Crawl4AI"]),
        ("10x", ["cheaper per answer", "than Claude crawling"]),
        ("MCP", ["Claude Code, Cursor,", "Claude Desktop"]),
    ]
    for i, (big, lines) in enumerate(stats):
        x = 96 + i * 370
        body += text(x, 462, big, ACCENT_B if i != 1 else "#A5A0FF", 56, 800)
        for k, line in enumerate(lines):
            body += text(x, 500 + k * 28, line, "#9198A1", 22)
    body += text(96, 598, "github.com/RayenMlayeh/jevcrawl · Apache-2.0", "#6E7781", 20)
    return svg(w, h, body, "jevcrawl")


def main() -> None:
    for name, t in THEMES.items():
        for stem, draw in [("logo", logo), ("scoreboard", scoreboard), ("cost", cost_per_answer),
                           ("crawl4ai", crawl4ai), ("how", how_it_works)]:  # fmt: skip
            (OUT / f"{stem}-{name}.svg").write_text(draw(t), encoding="utf-8")
    (OUT / "mark.svg").write_text(svg(120, 120, mark(0, 0, 120), "jevcrawl"), encoding="utf-8")
    (OUT / "demo.svg").write_text(demo(), encoding="utf-8")
    (OUT / "social.svg").write_text(social(), encoding="utf-8")


if __name__ == "__main__":
    main()

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.