arena.oxagen.sh

Your agent got worse last Tuesday. Nobody noticed.

Every leaderboard measures the model. Nobody measures your agent engine: the harness, the prompts, the tools, the loop you actually ship. Arena runs coding agents head-to-head on the same model, same budget, same timeout, grades them with held-out tests they can never see, and reports each metric separately with real statistics and full receipts.

git clone macanderson/arena Read the paper Read the spec

Report format shown with example data. Every real report carries its own intervals, p-values, and raw run directory.

The problem

Model benchmarks answer a question you are not asking

When your coding agent gets better or worse, the model is usually not what changed. You changed a system prompt, added a tool, reworked the planning loop, or bumped a dependency. Model leaderboards cannot see any of that. They hold the harness fixed and vary the model. Your job is the opposite: the model is a config value, and the engine is the product.

Arena isolates the engine. Two or more agents, one model, one budget, one timeout, the same tasks. If the numbers differ, the harness is the reason. That is the comparison that tells you whether your last sprint helped, and the one no public leaderboard runs for you.

Trust

Built to survive skeptical reading

The agent graded itself
Held-out verification. Tests live outside the workspace, are copied in only after the agent exits, and anything planted at the verify path is deleted first. CI proves every task fails pristine and passes the reference solution.
Different models or budgets
One config drives every agent. Unmatched runs are stamped matchedModels: false and the report leads with a warning banner.
Your harness broke the competitor
Harness failures are excluded. A CLI that cannot be invoked scores agent-error and is removed from every comparison. It never counts as the agent losing.
n = 1, no stats
Real statistics. Wilson 95% intervals, exact McNemar on paired outcomes, seeded paired-bootstrap CIs on speed, tokens, and cost. Same data plus same seed gives the same report, bit for bit.
Where is the raw data
Full receipts. Every trial writes its transcript, workspace diff, and a manifest with CLI versions, models, host, seed, and a one-line reproduce command.

Measurement

Four metrics, never blended

A single blended score hides the tradeoff you care about. Arena reports each metric separately, with its own interval, and refuses to average them.

accuracy
Held-out tests pass. Wilson 95% interval on every rate.
speed
Wall-clock, harness-measured. Medians with bootstrap CIs.
tokens
Normalized per adapter. Cache reads never inflate input.
cost
One shared price table, or no number at all. Never a guess.

CI integration

The regression gate: a tripwire for agent quality

Benchmarks are not only for bragging. Snapshot a good run as a baseline, commit it, and fail CI the day a change makes your agent less accurate, slower, or more expensive. With --require-significant, a drop only fails once it clears 95% interval noise, so small samples do not cry wolf. 

# establish a baseline once, commit it
pnpm arena run --agents mine --model anthropic/claude-sonnet-5 --trials 5
pnpm arena baseline save results/run-…        # writes arena-baseline.json

# in CI: rerun and gate. non-zero exit = regression
pnpm arena run --agents mine --model anthropic/claude-sonnet-5 --trials 5
pnpm arena gate results/run-… --require-significant

Go deeper

The paper, the spec, and the methodology

The State of Agent Benchmarking
A survey of the 2026 landscape: what exists, what it measures, and why almost none of it evaluates the engine you ship.
research paper
Agent Benchmark Protocol
A draft open standard: live tasks from real repos, verified fixes, golden traces, and grading any engine can plug into.
design spec · draft
Methodology
The contract behind any number Arena produces, including what it takes to publish an A-beats-B claim.
github.com/macanderson/arena