How well do AI coding agents work in your codebase?
Run agents against real bugs in your repo, in a sandbox on your own machine. Get pass rates, cost and regressions you can trust, without sending your code anywhere.
$
| agent | rate · cost | 95% CI | flags |
|---|---|---|---|
| agent-a | 0.71 · $0.38 | 0.62–0.79 | clean |
| agent-b | 0.64 · $0.12 | 0.55–0.73 | 1 regression |
| agent-c | 0.52 · local | 0.42–0.61 | 2 disqualified |
A leaderboard can't tell you if an agent will work for you
Leaderboards aren't your codebase
Public benchmarks use public repos. Your stack, conventions and test suite are different, and so are the results.
One run proves nothing
Agent output varies from run to run. A single pass or fail is noise. You need repeated runs and a confidence interval.
Agents can "pass" by editing tests
A green test suite means little if the agent weakened or deleted the test it was meant to satisfy.
Three steps, all on your machine
- 01
Point it at a repo
Pick your repo and the tasks to try: real bugs from your history, or tasks you write.
~/your-repoagentbench run --repo ./your-repo --runs 10
- 02
Agents run in a sandbox
Each agent attempts each task N times in an isolated sandbox with no network access.
~/your-repoagentbench run --repo ./your-repo --runs 10
- 03
Read the report
Pass rates with confidence intervals, cost, regressions and flagged runs, with every transcript attached.
~/your-repoagentbench run --repo ./your-repo --runs 10› report/index.html
A report you can make a rollout decision from
Everything you need to decide whether to roll out an agent, and which one.
Pass rate with confidence intervals
See when a difference between agents is real, and when it's run-to-run noise.
Cost and tokens
Spend per task and per success, so you can weigh accuracy against price.
Regressions
Catch fixes that break other tests, not only whether the target test passed.
Test-tampering detection
Runs that edit or delete tests are flagged and disqualified, not counted as passes.
Full transcripts
Every command, edit and test result, so you can see why an agent failed.
Per-repo profile
Which kinds of tasks agents handle well in this repo, and which they don't.
Your code never leaves your machine
Runs fully local with Ollama.
No cloud model needed.
The sandbox has no network.
Agents can't send your code out.
No telemetry.
Nothing is sent back to us.
Hosted models only if you choose.
Use Claude or GPT with your own API key when that's acceptable.
Numbers that mean what they say
Harness errors aren't model failures
If the sandbox or tooling breaks, the run is excluded, not counted against the agent.
No apples-to-oranges rankings
Repos tested with different models aren't ranked against each other.
Real-bug tasks are verified
Each task built from a real bug is checked in CI: the tests fail before the fix and pass after it.
Questions
No. Agents run in a local sandbox with no network access. With a local model through Ollama, nothing leaves your machine at all. If you choose a hosted model, prompts (including the code the agent reads) go to that provider under your own API key.
Local models through Ollama, or Claude and GPT with your own API key.
Leaderboards measure agents on public repos, usually with one run per task. AgentBench measures them on your repo, repeats each task several times, and reports confidence intervals so you can tell real differences from noise.
Not yet. It's in private beta now, and we plan to open-source it after v1.
Find out which agent works in your codebase
Tell us about your team and what you'd benchmark. We're letting people in in small groups.