The Research
Catch the bugs your tests can't see.
AI writes a third of new production code. The bugs it ships are the silent ones.
ReGrade pays for itself catching the first bug. ReGrade catches up to +13 extra bugs per trial — every model in our core lineup improved — across 28 AI models from 12 labs, at a marginal cost between negative (it pays for itself) and ≤16¢ per extra bug caught — versus an industry-typical $1,500–$50,000 per bug that ships to production.
+1.7 to +13
extra bugs caught per trial — every model in the core lineup improved
+825%
biggest single-model improvement (Haiku 4.5)
18/18
bugs fixed by every top-tier model with ReGrade
σ→0
same answer every run on Sonnet, Opus, GPT-5.5, Qwen3.6-Plus
600
trials in our pre-registered head-to-head test
28 / 12
AI models / labs tested
📊 The test, by the numbers.
ReGrade is a behavioral-diff context block your AI coding agent reads alongside its existing prompt — it shows the agent how the new code's runtime behavior differs from the old, the way a human checks for regressions at code review. No model retraining. No changes to your build or CI pipeline.
What we tested. Each trial = one AI coding agent (running autonomously, the way Claude Code or Codex CLI operate — shell tools, file access, code edits) attempts to fix 18 known bugs planted in a 300,000-line Python codebase. 1,700+ trials across 28 AI models from 12 labs — a core 17-model agent-mode lineup plus the newest releases across every major lab, US and non-US.
28
AI models
12
labs
1,700+
trials
18
known bugs
300,000-line Python
codebase
+825%
biggest gain
🛠️ Works with every major coding agent and model API.
3 agent CLIs tested · 17 LLMs across 6 providers.
Coding-agent CLIs
Claude Code
- Haiku 4.5
- Sonnet 4.6
- Opus 4.6
- Opus 4.7
Codex CLI
- GPT-5.4
- GPT-5.5
qwen-code
- Qwen3-Coder
- Qwen3-Coder-Plus
- Qwen3.6-Plus
- Qwen3.6-Max-Preview
Models tested via API
Gemini
- Gemini 3 Flash Preview
- Gemini 3.1 Pro Preview
Grok
- Grok 3 Mini
- Grok 3 Fast
- Grok 4.20 Reasoning
DeepSeek
- V4 Pro
- V4 Flash
One ReGrade context block in the agent's prompt. No model retraining. No changes to your build or CI pipeline. Tested working across 28 models from 12 labs, US and non-US.
🎯 Every model improves. No exceptions.
+33% to +825% bug-fix improvement across the lineup. Without ReGrade, the best model fixes only 13.3 / 18. With ReGrade, four models hit the ceiling and every other one moves up.
| Provider | Model | Without | With | Δ |
|---|---|---|---|---|
| Anthropic ★ | Fable 5 (Mythos-class) | 11.5 / 18 | 18.0 / 18 | +57% |
| Anthropic | Opus 4.7 | 8.3 / 18 | 18.0 / 18 | +117% |
| Anthropic | Sonnet 4.6 | 5.0 / 18 | 18.0 / 18 | +260% |
| OpenAI | GPT-5.5 | 13.3 / 18 | 18.0 / 18 | +35% |
| OpenAI | GPT-5.4 | 13.3 / 18 | 17.7 / 18 | +33% |
| DeepSeek | DeepSeek V4 Pro | 5.3 / 18 | 16.2 / 18 | +206% |
| xAI | Grok 4.20 Reasoning | 3.7 / 18 | 15.7 / 18 | +324% |
| Gemini 3.1 Pro Preview | 8.7 / 18 | 14.3 / 18 | +64% | |
| Alibaba | Qwen3.6-Max-Preview | 5.7 / 18 | 12.8 / 18 | +125% |
| Moonshot ★ | Kimi K2.6 (native CLI) | 6.2 / 18 | 16.1 / 18 | +160% |
Bug-fix counts from blind agent-mode evaluation (n=3–4 per model; the five frontier subscription cells are confirmed at n=30 in the pre-registered head-to-head below). ★ Fable 5 (Mythos-class, n=16) is one of the most advanced models we tested — even it left a third of the bugs unfixed on its own, and ReGrade took it to a perfect 18/18. ★ Kimi K2.6 is a preliminary n=8 cell, engaged-subset means.
🌍 The newest labs, too.
The behavioral-diff effect isn’t a quirk of the big US providers. We ran the same test on the newest releases from five more labs — each at 30 trials per model — and the rescue held every time, statistically decisive in each case.
| Lab | Model | Without | With | Δ |
|---|---|---|---|---|
| ByteDance | Seed-2.0-Lite | 2.9 / 18 | 14.7 / 18 | +407% |
| MiniMax | M2.5 | 2.1 / 18 | 10.9 / 18 | +419% |
| Zhipu | GLM-5.1 | 4.5 / 18 | 17.6 / 18 | +293% |
| StepFun | Step 3.7-flash | 3.4 / 18 | 11.0 / 18 | +221% |
| Mistral | Devstral 2 | 5.4 / 18 | 13.0 / 18 | +141% |
28 models across 12 labs, US and non-US. Every one of them fixed more silent regressions with ReGrade.
✅ The gains are real — proven, not assumed.
In a 600-trial head-to-head test whose design we specified in advance (so the results couldn't be cherry-picked after the fact), ReGrade beat all three comparison conditions — across every model tested.
| Control condition | Bugs fixed | vs ReGrade |
|---|---|---|
| No extra context (just the test suite) | 8.6 / 18 | 16.9 / 18 |
| Random gibberish, same length as ReGrade | 8.7 / 18 | 16.9 / 18 |
| Unrelated source code, same length as ReGrade | 8.4 / 18 | 16.9 / 18 |
What moves the needle is the behavioral signal in ReGrade, not the volume of extra context. Less than 1-in-a-billion-trillion chance the difference is a fluke.
🕵️ It catches what your tests can't.
Functional tests check status codes and required fields. They routinely miss the behavioral drift that actually breaks production: a CORS header that quietly changed, a list that's now an object, a timestamp format that broke a downstream consumer. ReGrade compares runtime behavior directly, so the agent sees the drift the test suite waved through.
35%
of AI patches that passed every functional test still silently broke behavior
191 of 548 test-passing patches, across 14 of 26 tasks
0 → 9
multi-file bug sites fixed — weaker models fixed 0 of 9 on their own
with ReGrade, agents clear all 9 sites of a cross-file regression
18
classes of silent drift planted — headers, encoding, ordering, response shape
the behavioral defects that pass type-checks, lint, and existing tests
Your test suite says “green.” ReGrade says “but the API’s behavior drifted.” That gap is exactly where production incidents come from.
🔎 Better answers, not just better detection.
Finding a bug is only half the job — a reviewer still has to understand and trust the fix. Even when a model already spots a regression, ReGrade’s behavioral evidence helps it explain exactly what changed and where.
2×
more precise bug explanations on a production codebase (Ghost CMS)
explanation precision rose from 0.40 to 0.91
9 of 10
models explained the bug better on a 291,000-line codebase (NetBox)
+3.2 points on a 20-point rubric — even where the model already found it
The difference between “something looks off” and a root-cause a human can act on with confidence.
⚡ Speed: 20–84% less real-world time per trial.
The ReGrade context cuts down on the back-and-forth where the agent runs shell commands and re-reads files trying to figure out what changed — the agent gets to the answer faster.
| Provider | Model | Without | With | Speed-up |
|---|---|---|---|---|
| Alibaba | Qwen3.6-Max-Preview | 50m 7s | 8m 4s | 84% faster |
| Gemini 3.1 Pro Preview | 10m 22s | 6m 46s | 35% faster | |
| OpenAI | GPT-5.4 | 4m 1s | 2m 48s | 30% faster |
| OpenAI | GPT-5.5 | 5m 17s | 4m 13s | 20% faster |
| Alibaba | Qwen3.6-Plus | 25m 26s | 21m 25s | 16% faster |
Bonus: in four models (Sonnet 4.6, Opus 4.7, GPT-5.5, Qwen3.6-Plus) every trial produces the identical outcome — trial-to-trial randomness drops to zero. No more flaky CI retries.
💰 Cost per bug drops 23–89% on top-tier models.
Adding ReGrade reduces LLM input/output cost per trial AND finds more bugs, so the total cost per bug fixed is meaningfully lower with ReGrade than without it on most top-tier models. Even on the four models where it doesn't pay for itself outright, the marginal cost stays under ≤16¢ per extra bug caught — versus an industry-typical $1,500–$50,000 per bug that ships to production.
| Provider | Model | Without $/bug | With $/bug | Change |
|---|---|---|---|---|
| Alibaba | Qwen3.6-Max-Preview | $0.333 | $0.036 | -89% |
| xAI | Grok 4.20 Reasoning | $0.235 | $0.077 | -67% |
| OpenAI | GPT-5.5 | $0.156 | $0.091 | -42% |
| Anthropic | Sonnet 4.6 | $0.126 | $0.092 | -27% |
| OpenAI | GPT-5.4 | $0.020 | $0.015 | -25% |
| Anthropic | Opus 4.7 | $0.257 | $0.199 | -23% |
Six of eight top-tier models get cheaper per bug with ReGrade, with reductions from −23% to −89%.
🧹 Fixes more — and cleans up after itself.
A tool that fixes bugs but quietly breaks other things is not a safety win. So we measured the other side of the ledger: how many clean, working fields the agent damaged while making its fix. In our pre-registered head-to-head (n=30 per model), the behavioral diff cut collateral damage to near zero on every capable model.
| Provider | Model | Clean fields broken | With ReGrade |
|---|---|---|---|
| OpenAI | GPT-5.4 | 29.4 | 4.0 |
| OpenAI | GPT-5.5 | 19.6 | 0.0 |
| Anthropic | Opus 4.7 | 5.8 | 0.3 |
| Anthropic | Opus 4.8 | 5.8 | 0.0 |
| Anthropic | Sonnet 4.6 | 2.6 | 0.3 |
Mean clean, v1-matching response fields the agent broke per run — lower is safer.
The behavioral safety net cuts both ways: more of the real bugs caught, far fewer new ones introduced.
The honest caveat: this is strongest on the models with the highest baseline error rates. On a few high-rescue models, more fixing comes with more editing, and collateral changes can rise — a trade-off we report in full in the paper. The clean, controlled result above is the claim we stand behind.
🏆 Three recommended models, ranked by cost with ReGrade.
For teams choosing their AI coding agent today: best results-per-dollar with ReGrade.
🥇
Gemini 3 Flash Preview
17.7 / 18
bugs fixed
$0.008
per bug
🥈
GPT-5.4
17.7 / 18
bugs fixed
$0.015
per bug
🥉
DeepSeek V4 Pro
16.2 / 18
bugs fixed
$0.017
per bug
Five more model + ReGrade combinations clear the cheap-AND-effective threshold. Works equally well across providers.
ReGrade is a single addition to the agent's prompt — you pay only the marginal LLM tokens for that context, with no separate ReGrade fee per bug.
Want ReGrade for your AI coding agent?
Drop the context block into your agent's prompt. No retraining. No CI changes.