Benchmark coding models on DeepSWE via OpenRouter

Benchmarks a model on DeepSWE, hosted at deepswe.datacurve.ai, a 113-task Harbor-compatible coding-agent benchmark. It runs via Pier, a fork of Harbor, which in turn drives mini-swe-agent, a coding agent built to be model-agnostic rather than tied to one vendor. Because mini-swe-agent is model-agnostic and Pier just orchestrates it against the task corpus, any model reachable through OpenRouter can be scored on the same 113 tasks without writing a bespoke integration for that model first. The output is meant to be a reproducible, repeatable evaluation plan with concrete artifacts on disk, not a one-off manual comparison: the same commands, run again later, should reproduce the same score for the same model, which matters when comparing multiple candidate models over time or re-checking a score after a model update.

Use it when the user wants to benchmark a model on DeepSWE or mini-swe-agent tasks specifically, or needs a reproducible coding-agent evaluation plan with concrete output artifacts to point at afterward, rather than an ad-hoc manual comparison that can't be rerun the same way twice. Before touching any of the run commands, state-check the prerequisites rather than assuming they're already satisfied: confirm the uv, git, and docker binaries are actually installed, confirm the Docker daemon is actually running rather than just installed, since Pier sandboxes every task in Docker by default, and confirm an OpenRouter API key is already present in the environment. A short state-check command sequence covers all three:

which uv git docker || echo "MISSING: install uv, git, docker"
docker info >/dev/null 2>&1 || echo "MISSING: Docker daemon not running (Pier's default sandbox)"
echo "OPENROUTER_API_KEY set? ${OPENROUTER_API_KEY:+YES}"

A cloud sandbox flag exists as an alternative to the Docker default for teams that would rather not run task sandboxes locally. The OpenRouter key specifically must already be present in the environment before starting; if it's unset, the correct move is to ask the user to configure their own preferred secret-management path, never to read shell startup files looking for it, never to print a discovered secret back to the user, and never to invent a placeholder key just to get a command to run.

Every trial's output lands in a per-run, per-trial directory on disk, and that output can be inspected three different ways depending on how much detail is needed: a quick view of a single run, an aggregate analysis across all trials in a run for overall scoring, or a deeper per-task critique when a specific result needs scrutiny rather than just a pass or fail number. Whatever the outcome, the resulting report should always state the exact command that was actually run, whether it passed or failed, the score itself, and any blockers hit along the way, so the run can be reproduced or debugged later without guessing what was actually executed. Each documented failure mode has its own specific fix rather than a generic "just retry": an HTTP 401 means the OpenRouter key needs to be re-exported, since it's either missing or wrong; a missing-provider error means the model slug is missing its routing prefix, fixed by switching to the fallback route or adding the explicit OpenRouter model-class flag to the preferred route; a model-not-mapped cost error is fixed by explicitly telling the cost tracker to ignore that class of error; and an unrecognized flag almost always means the installed Pier version has drifted from whatever documentation is being followed, resolved by checking the CLI's own current help output rather than trusting older instructions.

Anyone evaluating a model's coding-agent performance who wants a reproducible, OpenRouter-driven DeepSWE benchmark run with clearly separated smoke-test, deterministic-subset, and full-corpus stages, plus concrete, specific fixes for the most common OpenRouter wiring failures instead of generic trial-and-error debugging when a run doesn't behave as expected.