A benchmark on my own code, that corrects itself in public

Whose code. A SWE-bench score says nothing about how a model behaves in a specific Go, Python or TypeScript codebase with its own conventions. So the harvester walks this machine’s own git history for commits that touch both a test file and a source file and are small enough to be a fair single-shot task. The model is measured on the code it will actually be asked to work on.

How many times. Public benchmarks run each task once and report a single pass or fail. That hides a real failure mode: a model that passes a task once and fails it twice more is not “80% reliable” in any useful sense when you only get one shot in production. Every model-task pair runs three times here, and the report carries an agreement rate and a bootstrap confidence interval, so a gap between two models can be called noise or called real instead of asserted from one sample.

Fifteen tasks a cheaper model had already failed, each run twice per tier, every pass integrity-verified: the test had to fail before the patch and pass after it, with no writes to the test file. The small tier solves two thirds of what the large tier does at a quarter of the price per success, which is a routing decision rather than a ranking.

The dollar levels are inflated — a large system preamble rides on every call — but it rides on all three equally. Ratios hold; levels do not. A benchmark that only reported the flattering half of that would not be one.

One task failed for every tier: zero passes out of two, three times over, with no timeouts and no variance. Zero spread across three capability tiers is a signature of a broken task, not a hard one — a genuinely hard problem separates a large model from a small one.

It was the harness. The graded test was asynchronous, the repository declared its async plugin as an optional extra, and the command the benchmark ran did not install extras. So the test could not execute at all. The decisive check cost no model calls: check out the real fix commit — the change that is the answer — and run the graded command against it. It scored zero. No patch any model could write would have passed.

Three tiers had been reported a full capability step lower than they were, on the strength of a task nothing could pass. The task was repaired and re-run rather than dropped, and the findings file now carries each wrong figure inline beside the one that replaced it.

Rejecting public benchmarks for scoring models on somebody else’s code has an honest consequence, and it applies here first: a score from this project is a statement about one model, one corpus, one revision and one date. It is not a leaderboard entry, and two scores from different repositories do not rank the models against each other.

That is not a gap waiting to be closed. A benchmark whose numbers were comparable across corpora would have had to abstract away the codebase — its conventions, its test shapes, its commit habits — and the codebase is the entire thing being measured.

The corpora are already different instruments before a single model is called. Across a hundred repositories on two machines, the share of a repository’s history that qualifies as a task ranges from 4.0% to 74.5% among the thirty that yield anything at all. One repository measured 60.8% when its corpus was first built and 40.6% under identical criteria today — twenty points of drift from history growth alone, most of it automated release commits entering the denominator.

• Patch — coding ability, graded by running the model’s fix against a real test suite.
• Reading — effective context length, graded by exact retrieval of a real fact planted in real repository content.
• Discovery — bug finding, graded by whether the reported line range intersects the real fix.
• Scout — symbol-location accuracy, run through a real tool loop rather than a plain chat completion, and graded against a deterministic symbol index.

Two failure modes are scored as real failures rather than retried away: determinism decays with output length, so identical prompts stop returning identical prose past a few hundred tokens; and a reasoning model can return an empty response when its whole token budget went to reasoning it never shows you. A retry would have hidden both.