Benchmark coding models on DeepSWE via OpenRouter

Use this guide when a team needs to evaluate a coding agent against a fixed software engineering benchmark and produce reviewable run artifacts. It explains how to check the local toolchain, connect a supported model provider, validate one task, choose a deterministic subset, and inspect results. It is intended for evaluation planning and controlled execution, not for silently launching a large or paid run.

The documented workflow uses uv, Git, Docker, a benchmark runner, and a task agent. Check that the required executables are installed and that the container service is available before setup. The runner creates an isolated sandbox for each task by default, so a running Docker daemon is part of the ordinary local path. A cloud execution option is described, but it depends on separately configured access and should only be selected when the environment is ready.

Model access is provided through an environment variable. Confirm only whether the variable is present; never print its value or copy it into logs. If it is absent, stop and have the environment owner configure the approved secret-management path. Do not place credentials in command history, task artifacts, or reports.

Clone the benchmark repository and install its runner with the preferred package tool, or use the documented source installation alternative. Run commands from the benchmark repository root and use paths relative to that checkout. The runner bundles the task agent as its driver. One documented provider route uses the provider's native model class and exact model slug; a fallback route uses a provider-prefixed model name. Verify the chosen model identifier against the provider's model catalog, and check the installed runner's help output because flags can vary between versions.

Start with one known task. Confirm that the run completes, the agent emits actions rather than an authentication or format error, and the runner saves a score or trajectory. Treat authorization failures as a configuration issue; treat provider mapping errors as a routing or model-name issue. Correct the wiring and repeat the single-task check before selecting a subset. Avoid launching a full corpus merely to discover whether credentials or flags work.

For a repeatable subset, use the runner's task-count and sample-seed options. Record the exact command, runner version, model slug, seed, selected task count, and pass/fail result so another evaluator can reproduce the configuration. Full-corpus runs can consume substantial time and provider resources; scope them deliberately and use the documented confirmation process for that environment.

The runner stores trial artifacts beneath a run-specific jobs directory. Use its viewing and analysis commands to inspect trajectories, aggregate results, and identify failures. A useful report names the exact invocation, evaluation scope, outcome, score when available, and any environmental blockers. Distinguish a completed run from a successful benchmark result, and avoid claiming comparative performance without a comparable baseline and matching configuration.

The source also documents failure patterns such as authentication rejection, missing provider mapping, unknown model cost metadata, and version-specific flags. Diagnose each from its actual error and make the smallest configuration change. Some models may lack cost metadata; any option that suppresses cost-tracking errors changes accounting behavior and should be explicit in the run record.

This workflow adapts a community guide, so confirm local paths, tool versions, provider access, and task-agent features before relying on examples. Commands may access remote services, schedule work, automate a browser, or write files. Confirm the target environment and authorization before those actions. Keep credentials out of artifacts, and do not submit results to an external leaderboard or contact an address unless that destination and publication are explicitly authorized. The guide is most useful for controlled, auditable benchmark runs with a small smoke test followed by a deliberately scoped evaluation.