Benchmark LLM agents on fixing real-world CVE vulnerabilities

CVE-Bench

A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities. Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite.

---

Requirements

• Python 3.12+
• Docker
• OPENAI_API_KEY, ANTHROPIC_API_KEY, and/or POOLSIDE_API_KEY in your environment (or a .env file)

Install dependencies:

pip install poetry
poetry install

---

Task structure

Each task lives under tasks/{CVE-ID}/ and contains:

tasks/CVE-2026-33175/
├── meta.json           # GHSA ID, CWE, CVSS, repo URL, vulnerable and fixed SHAs
├── setup.sh            # Clones repo, checks out the vulnerable SHA, installs dependencies
├── run_tests.sh        # Injects test_security.py into the repo and runs pytest
├── test_security.py    # Security tests (xfail on vulnerable code, pass on the fix)
├── advisory.md         # Full GHSA advisory (richest prompt)
├── diagnose.md         # Behavioural description only — no file or function names
├── locate.md           # File and function only — no description of the flaw
└── Dockerfile          # Optional; only present when the task needs extra system deps

meta.json example:

{
  "ghsa_id": "GHSA-xxxx-xxxx-xxxx",
  "cwe": ["CWE-287"],
  "cvss": 9.1,
  "repo": {
    "url": "https://github.com/org/project",
    "vulnerable_sha": "abc123^",
    "fixed_sha": "abc123"
  }
}

setup.sh is idempotent and safe to re-run. test_security.py is kept hidden from the agent during the run and injected only after the agent finishes.

---

Building Docker images

python build.py

This builds:

1. A shared base image (cve-bench/base) - Python 3.12, git, poetry, and the harness.
2. One task image per task (cve-bench/{task-id}) - extends the base, copies the task directory, and runs setup.sh.

Options:

# Build specific tasks only
python build.py --task CVE-2026-33175 CVE-2026-42561

# Skip rebuilding the base image
python build.py --skip-base

Task images are built in parallel (up to 5 workers). If a task directory contains a Dockerfile, it is used instead of the generic docker/task.Dockerfile.

---

Validating tasks

Before running the benchmark, verify that each task's security tests correctly distinguish vulnerable from fixed code:

python validate.py

For each task, this runs three phases inside the task container:

Phase	What it checks
vulnerable	Security tests must fail (or xfail) on the vulnerable SHA
fixed	Security tests must pass on the fixed SHA
regression	Non-security tests must pass on the fixed SHA

Results are displayed as a live table. Exit code is 1 if any task fails any phase.

# Validate specific tasks only
python validate.py --task CVE-2026-33175 GHSA-r758-8hxw-4845

# Skip rebuilding images before validation
python validate.py --skip-build

---

Running the benchmark

python benchmark.py --model openai:gpt-5.5 poolside:laguna-m.1 --prompt-type advisory

Options:

Flag	Description	Default
--model	One or more provider:model-id strings	all configured models
--prompt-type	advisory, diagnose, locate, or any combination	all three
--task	One or more task IDs	all tasks
--clean	Delete existing results for the selected scope before starting	off

Supported providers:

Provider	Format	API key env var
OpenAI	openai:gpt-5.5	OPENAI_API_KEY
Anthropic	anthropic:claude-haiku-4-5-20251001	ANTHROPIC_API_KEY
Poolside	poolside:laguna-m.1	POOLSIDE_API_KEY

Each run produces a JSON result file in results/:

results/{task-id}__{provider}:{model}__{prompt-type}.json

Existing result files are skipped automatically. Runs execute concurrently across tasks (up to 20 workers), with per-provider rate limiting (one active request per provider at a time) to avoid 429s.

---

Result format

Each result file is a JSON object with the following structure:

{
  "cve_id": "CVE-2026-33175",
  "model_id": "openai:gpt-5.5",
  "prompt_type": "advisory",
  "timestamp": "2026-05-01T12:00:00",
  "model_duration_s": 142.3,
  "test_duration_s": 8.1,
  "turns": [
    {
      "tool_calls_and_results": [...],
      "input_tokens": 12400,
      "output_tokens": 310
    }
  ],
  "tests": [
    {
      "kind": "security",
      "name": "test_email_verified",
      "outcome": "passed"
    }
  ]
}

tests[].kind is either "security" (from test_security.py) or "regression" (from the project's own test suite). A run is considered solved only if all security tests pass and no regression tests fail.

---

Generating charts

python generate_charts.py

Reads all result files from results/ and writes SVG charts to docs/images/charts/. Requires Chrome/Chromium for Bokeh's headless export (via chromedriver-binary).

---

Harness architecture

The harness runs inside each Docker container as python -m harness.run. It is responsible for loading the prompt, running the agentic loop, and writing the result file.

src/harness/
├── run.py                  # Entry point; parses args, wires components, calls BenchmarkRunner
├── client/
│   ├── factory.py          # Parses provider:model-id, returns the correct LLMClient
│   ├── _client.py          # Abstract LLMClient, ToolCall and LLMTurn dataclasses
│   ├── anthropic.py        # Anthropic SDK integration
│   └── oai.py              # OpenAI SDK integration (also used for Poolside)
├── agent/
│   ├── core.py             # Agentic loop: calls client, dispatches tool calls, threads messages
│   └── runner.py           # Wraps Agent, tracks timing and turn list
├── bench/
│   ├── runner.py           # Orchestrates setup → agent → security tests → regression tests
│   ├── result.py           # BenchmarkResult and TestResult dataclasses, JSON serialisation
│   └── repository.py       # Writes result files to disk
└── task/
    ├── tools.py             # Tool implementations: ListFiles, ReadFile, SearchInFiles,
    │                        #   EditFile, CreateFile, DeleteFile, RunPytest
    └── prompt_loader.py     # Reads advisory.md / diagnose.md / locate.md

Tools available to the agent:

Tool	Description
list_files	List files and directories in the repository
read_file	Read file contents, optionally a line range
search_in_files	Regex search across the codebase with optional file glob
edit_file	Replace a range of lines in an existing file
create_file	Create a new file
delete_file	Delete a file
run_pytest	Run the project's test suite; returns a JSON report

All tools validate paths against the repository root to prevent directory traversal. The agent does not have access to test_security.py or to the git history.

The agent loop runs for at most 20 turns. If the turn ceiling is reached, the run is recorded as-is and the security tests are still executed against whatever state the agent left the repository in.

---

Adding a task

1. Create tasks/{CVE-ID}/ and add meta.json, setup.sh, run_tests.sh, test_security.py, advisory.md, diagnose.md, locate.md.
2. Make setup.sh and run_tests.sh executable (chmod +x).
3. Validate: python validate.py --task {CVE-ID}.
4. Build: python build.py --task {CVE-ID}.

---

Disclosure

This work was conducted as independent research. At the time of conducting the research and preparing this repository, I had no institutional affiliation.

---

Citing this work

@misc{gattipinheiro2026cvebench,
  author       = {Gatti Pinheiro, Giovanni},
  title        = {{CVE-Bench}: Benchmarking {LLM} Agents on Real-World Security Vulnerability Fixes},
  year         = {2026},
  howpublished = {\url{https://giovannigatti.github.io/cve-bench}},
  note         = {Code available at \url{https://github.com/GiovanniGatti/cve-bench}}
}

---