The autonomy flag is part of the experiment, not a footnote: running one agent at full autonomy against another at its default measures permission defaults rather than agents.
agent
version
model
autonomy
freecode
0.30.1
minimax/MiniMax-M3
--agent danger: no permission prompts
claude-code
2.1.251 (Claude Code)
MiniMax-M3
--dangerously-skip-permissions: all permission checks bypassed
Resolved
Marked resolved by the official SWE-bench grader. Computed over the 10 instances every agent in this matchup ran.
freecode77%
77%23/30 trials
claude-code80%
80%24/30 trials
freecode
23/30
claude-code
24/30
Median wall time
Spawn to agent exit. Shorter is better — the longest bar is the slowest agent, not the winner. An agent that runs the tests pays for it here, which is not obviously a vice.
freecode76s
76smedian of 30
claude-code64s
64smedian of 30
Where the time goes
freecode~20 turns per trial·~3.8s per turn
claude-code~20 turns per trial·~3.2s per turn
Wall time is turns × time per turn: an agent can lose this bar by making slow turns, or by making more of them. The split above says which.
Slowest median in this matchup: 76s.
Median patch size
Bytes of diff produced. A fact about each harness's style, not a score — a bigger patch is not a better fix, and a smaller one is not automatically surgical.
freecode659B
659Bmedian of 30
claude-code642B
642Bmedian of 30
New files created
None — across all 60 trials, every patch edited existing code only. That is what a bug-fix task set should look like: a harness that scaffolds new files here would be a smell, not a feature.
Tokens per trial
Input + output counted off one recording proxy for every agent — the same meter, the same rules, never each vendor's own accounting. Fewer is cheaper, but an agent that gives up early is also cheap; read this next to the outcome bar, not instead of it.
freecode287k
287kmean of 30 metered trials · ~20 turns
claude-code710k
710kmean of 30 metered trials · ~20 turns
Prompt cache
How much of each trial's input the provider served from its prompt cache, off the same meter as the token chart. The pale segment was read from cache at a discounted rate; the solid segment is fresh context, billed in full. Bars share one scale, so their lengths also compare input volume.
freecode94.4% cached
94.4% cached~16k fresh of 282k/trial
claude-code91.6% cached
91.6% cached~59k fresh of 696k/trial
read from cache (discounted)fresh input (billed in full)
A high hit rate means the harness keeps its prompt stable enough for the provider to reuse it. What each trial actually pays full price for is the solid segment: freecode ~16k fresh input tokens per trial vs claude-code ~59k — the hit rate is where a token gap becomes a cost gap.
Cost per trial
USD from a committed rate card applied to the proxy's counts — identical pricing for every agent. Cache reads are discounted, not added. The worst-single-trial figure is there because a good mean can hide one runaway.
freecode$0.0262
$0.0262worst single trial $0.1252
claude-code$0.0728
$0.0728worst single trial $0.4438
On the 8 bugs every agent solved: freecode $0.0138 (147k tok) · claude-code $0.0265 (257k tok). Averaging over failures would make the quitter cheapest — this mean covers solved bugs only.
Cost distribution
Every metered trial, one dot, by bug. A mean hides the outliers — this does not. An open ring is a trial that ran but did not resolve, so a cheap dot low on the chart is not always a win.
freecodeclaude-codeopen ring = not resolved
What each fix cost
Only the 8bugs every agent resolved (spec §7.2) — same bug, same model, so the gap is how much context each harness moved to get there. Cost is the mean over that agent's resolved trials of the bug; the cheaper agent on each row is in bold.
Set default FILE_UPLOAD_PERMISSION to 0o644.django-10914
1.7× cheaper1.9× fewer tokens
freecode
$0.0196 (205k)
claude-code
$0.0336 (383k)
Allow FilePathField path to accept a callable.django-10924
1.1× cheaper1.1× more tokens
freecode
$0.0292 (329k)
claude-code
$0.0334 (287k)
Incorrect removal of order_by clause created as multiline RawSQLdjango-11001
2.6× cheaper2.0× fewer tokens
freecode
$0.0187 (213k)
claude-code
$0.0488 (429k)
sqlmigrate wraps it's outpout in BEGIN/COMMIT even if the database doesn't support transactional DDLdjango-11039
2.1× cheaper1.9× fewer tokens
freecode
$0.0098 (101k)
claude-code
$0.0206 (193k)
Correct expected format in invalid DurationField error messagedjango-11049
1.4× cheaper1.3× fewer tokens
freecode
$0.0147 (158k)
claude-code
$0.0206 (213k)
UsernameValidator allows trailing newline in usernamesdjango-11099
delete() on instances of models without any dependencies doesn't clear PKs.django-11179
3.5× cheaper2.8× fewer tokens
freecode
$0.0046 (38k)
claude-code
$0.0161 (108k)
mean over 8 bugs
1.9× cheaper1.7× fewer tokens
freecode
$0.0138 (147k)
claude-code
$0.0265 (257k)
Per instance
One row per bug, one cell per trial. Disagreement between trials of the same agent is the interesting signal — it is what a single-run benchmark cannot show you.
resolvedpatched, not resolvedno patch
Set default FILE_UPLOAD_PERMISSION to 0o644.django__django-10914
Published rather than managed. A benchmark whose limitations live in a footnote is an advertisement.
10 instances is a demo, not a leaderboard
SWE-bench Lite is 300 instances across 11 repositories. One repository's idioms are not the field, and a handful of instances from it is an anecdote with decimal places.
Contamination is unfixable on this task set
Every SWE-bench Lite fix is public and predates the training cutoff of the models involved. Closing the network stops lookup, not recall. That is survivable for a relative comparison — every agent gets the same unfair advantage — but it invalidates any absolute reading.
This compares harnesses, not models
Every agent is pinned to the same model (MiniMax-M3) on the same key, and each keeps its own system prompt. Change the model and it becomes a different experiment with the same table.
We built the harness and we are in the table
Every incentive here points one way. The countermeasures are structural: the task set is external, the grader is external, the artifacts are published, and a loss goes in the headline.
How it ran
The whole pipeline, in order. Every step is the same for every agent — the moment one step differs per agent, the benchmark stops comparing harnesses and starts comparing our treatment of them.
1
Pick the bugs
10 real django/django issues from SWE-bench Lite, fetched from HuggingFace. The gold patch, the test patch, and the maintainer hints are stripped before anything touches disk — the answer key never enters this repo.
2
Build the workspace
Each trial gets its own fresh checkout at the commit just before the real fix landed, cloned from a cached local mirror on a detached HEAD. No network clone in the timed path, nothing shared between trials.
3
Give every agent the same prompt
One string, identical for all agents: the upstream issue text, plus instructions not to touch tests and not to commit. The prompt lives in its own file so changing it reviews as what it is — a change to the experiment.
4
Pin the model, max the autonomy
Every agent runs MiniMax-M3 on the same API key, each with its own full-autonomy flag (see Setup above). Running one agent at full autonomy against another at its default would measure permission defaults, not agents.
5
Isolate each trial in Docker
Each trial runs in a container on an internal Docker network whose only exit is the metering proxy — the agent cannot look the fix up, and a fresh $HOME means nothing an agent learns carries into the next trial.
6
Meter everything through one proxy
All model traffic passes through a single recording pass-through proxy. Tokens are counted by that one meter and priced from a committed rate card — the same accounting for every agent, never each vendor's own dashboard.
7
Run the matrix, keep the diff
agents × bugs × trials, 3 trials per bug here. When the agent exits, its working-tree diff is extracted as the patch — that diff is the entire submission.
8
Grade with the official harness
The official SWE-bench harness applies each patch in its own Docker environment and runs the project's real test suite. Only its verdict marks a trial resolved — 'produced a patch' is never promoted to 'fixed the bug'.
9
Publish the evidence
Every trial writes its prompt, argv, patch, and stdout/stderr to the results directory, and the numbers land on this page by finishing. There is no editorial step between the run and what you are reading.
Check it yourself
The harness, the agent adapters, and the task list are all in the repo. A run lands on this page by finishing — there is no editorial step between the numbers and you.
Every trial writes its full evidence — prompt, argv, patch, stdout/stderr — to bench/agent-bench/results/<run>/. The task set is SWE-bench Lite, fetched from HuggingFace with the answer key stripped before anything touches disk.