---
title: "ESCAPES: a benchmark of 327 bugs that reached real users"
slug: escapes-a-benchmark-of-327-bugs-that-reached-real-users
url: https://listedarticles.com/articles/escapes-a-benchmark-of-327-bugs-that-reached-real-users
canonical_url: https://breken.ai/benchmark
content_type: research
language: en
published_at: 2026-11-14T11:13:44.585Z
updated_at: 2026-11-14T11:13:44.585Z
authored_by: human
publisher: "breken"
publisher_url: https://listedstartups.com/companies/breken
topics: ["benchmarks", "software-engineering", "ai-agents", "developer-tools"]
license: all-rights-reserved
word_count: 3592
reading_minutes: 16
citation: "breken. \"ESCAPES: a benchmark of 327 bugs that reached real users.\" 14 Nov 2026. https://breken.ai/benchmark (all-rights-reserved)"
# The full text follows. The web page shows an extract and sends readers
# to the source above; quote the citation and link the canonical URL.
---

# ESCAPES: a benchmark of 327 bugs that reached real users

> breken's open ESCAPES benchmark rewinds 33 open-source products to 1 July 2026 code and scores 235 user-reported bugs: breken's run-based system caught 76 (32%) with 20 false alarms versus 37 for Claude Fable 5.1 in Claude Code; vendor-run results with public data and judge prompts.

Benchmark·October 2026

# ESCAPES

327 bugs that passed code review and tests, reached real users, and were fixed. We rewound 33 open-source products to the code before each fix and asked one question: how many could have been caught before a user hit them?

Escapes caught before the user reported them

One scan per product, on its code as of 1 July 2026. 235 of the 327 bugs were already in that code and were reported by users in the weeks after. The share of those 235 each system had already found.

brekenruns the product

76 bugs · 32%

Claude Fable 5.1Claude Code

37 bugs · 16%

Claude Opus 5.5Claude Code

33 bugs · 14%

GPT-6 AstraCodex CLI

28 bugs · 12%

Static rulesSemgrep

9 bugs · 4%

**0****10****20****30****40% of 235**

breken found 76 of the 235, with 20 false alarms across the 33 products. The best model reading the code found 37, with 59 false alarms. Full table in section 03.

Contents

1. [01What escapes](#escapes)
2. [02How it was measured](#measured)
3. [03Catching them first](#results)
4. [04Four escapes](#cases)
5. [05What it means](#means)
6. [06Open benchmark](#open)
7. [07Full methodology](#method)

## 01What escapes

The bugs

**327**real, user-reported bugs from 33 open-source products

**152 days**median time a bug sat in the code before a user reported it

**53 days**median gap between the 1 July snapshot and the user's report

**71%**only appear when the product runs: data, users, failures or time

Every one of these bugs got through a project's own review and test suite. That is not a failure of review: most of them cannot be seen in a diff. They need a real data shape, a second user, a partner's outage or a month to pass. That is the layer ESCAPES measures.

**Caught, by what the bug needs**share of each row's bugs · 235 in all

Each bug is labelled by what it takes to see it. Per row: the share caught by the best model reading the code, Claude Fable 5.1 through Claude Code, and by breken running the product.

Claude Fable 5.1 reading the code (Claude Code)breken, running the product

Visible by reading one file69

*33%*

*32%*

Needs the product running51

*12%*

*39%*

Needs a real data shape42

*7%*

*33%*

Needs a second user or concurrency22

*5%*

*32%*

Needs an outside service to fail19

*5%*

*37%*

Needs time to pass (renewal, expiry, upgrade)17

*6%*

*24%*

Needs a specific environment15

*13%*

*13%*

**0****10****20****30****40%**

Reading the code and running the product find about the same share of bugs visible in one file. Beyond that, Claude Fable 5.1 finds 14 of 166; running finds 54.

## 02How it was measured

- **The bugs:** reported by a real user on the product's public tracker, fixed by a merged pull request between 1 July and 19 October 2026 (after every tested model's training cutoff), and present in the code when the user hit them.
- **The run:** the product's code and docs as of 1 July 2026, nothing later. The system reports the bugs it expects users to hit.
- **Why 1 July:** the first day after the training cutoff of every model tested, so none of them had seen these bugs or their fixes. One scan on that day is what a customer would get. 235 of the 327 bugs were already in the code by then; those are the ones scored. The other 92 arrived later.
- **A catch:** the finding describes the failure the user later reported. Wherever possible it is confirmed by running: its reproduction fails before the fix and passes after.
- **Checking:** an independent judge model, one that no system on the board runs on, grades the rest. People check every match, every near miss and a 20% sample.
- **Reference lines:** frontier AI models reading the code through their own coding agents (Claude Code, Codex CLI), with the same budget, to show what reading alone finds. Static rules (Semgrep) as the floor.
- **The score:** escapes caught, as a share of the 235. Beside it a weighted score, as in RECON: a catch earns one point, a false alarm loses one.

The full dataset rules, the exact prompt the agents get, the judge and every metric definition are in section 07.

## 03Catching them first

breken caught 76 of the 235 escapes, 61 of them confirmed by running. The 3 frontier models reading the code caught 28 to 37; static rules caught 9.

**Figure 1 · Weighted score by system**(caught − false alarms) ÷ 235

A catch earns one point and a false alarm loses one, divided by the 235 bugs. Below zero, a system raised more false alarms than bugs it caught.

breken · runs the productcaught 76 · false alarms 20

+23.8%

Claude Fable 5.1 · Claude Codecaught 37 · false alarms 59

−9.4%

Claude Opus 5.5 · Claude Codecaught 33 · false alarms 69

−15.3%

GPT-6 Astra · Codex CLIcaught 28 · false alarms 86

−24.7%

Static rules · Semgrepcaught 9 · false alarms 112

−43.8%

**−50%****−25%****0****+25%****+50%**

**Table 1 · Full results**one scan per product · one run per system

SystemWeightedCaughtBeyond 1 fileProvenFalse alarmsUnreportedPrecisionHead start

1brekenruns the product+23.8%76 · 32%54 of 1666120 · 0.63885%44 days

2Claude Fable 5.1Claude Code−9.4%37 · 16%14 of 166959 · 1.81346%39 days

3Claude Opus 5.5Claude Code−15.3%33 · 14%12 of 166769 · 2.11139%37 days

4GPT-6 AstraCodex CLI−24.7%28 · 12%10 of 166586 · 2.6930%34 days

5Static rulesSemgrep−43.8%9 · 4%0 of 1660112 · 3.4612%26 days

**Caught:** escapes caught, of the 235. **Beyond 1 file:** caught among the 166 that need the product running. **Proven:** catches with a reproduction that fails before the fix and passes after; the rest judged, then confirmed by a person. **False alarms:** total, then per product. **Unreported:** real defects no user had reported yet. **Precision:** real findings ÷ all findings. **Head start:** median days between the scan and the user's report, over caught bugs.

## 04Four escapes

Three catches and one miss.

### An e-commerce platform: a refund after a partial captureCaught 47 days before the user

Web app · needs a real data shape · proven by running

The userRefunding an order after a partial capture refunded the full authorised amount.

What it neededA partial-capture state. The change that introduced the bug never touched it.

What breken didRan checkout with a partial capture, then refunded. The ledger showed the full amount; the same steps without the partial capture were correct.

Proof**Proven by running.** Its reproduction failed before the fix and passed after.

### A storage sync CLI: a connection dropped mid-transferCaught 63 days before the user

CLI · needs an outside service to fail · proven by running

The userA transfer cut off by a dropped connection was reported as complete. The half-copied file stayed, marked as synced.

What it neededA network drop at the wrong moment, then a second sync.

What breken didCut the connection mid-transfer, synced again, compared checksums on both sides. They differed.

Proof**Proven by running.** Two commands reproduce it.

### A collaborative editor: two editors and a reconnectCaught 29 days before the user

Web app · needs a second user and a failure · judged, confirmed by a person

The userAfter a short disconnect, one person's edits overwrote another's.

What it neededTwo people on one paragraph and a dropped connection at the same moment.

What breken didTwo simulated editors worked on one paragraph while one connection dropped and came back. An edit was lost in 3 of 3 runs, never without the drop.

Proof**Judged, then confirmed.** Two live sessions cannot run in the matcher; the judge said match and a person agreed.

### A miss: billing for streamed tool calls in an AI gatewayNot caught

API gateway · needs the product running · missed

The userStreamed responses with tool calls were billed twice for the tool output.

What it neededStreaming and tool calls together, on a billed account.

What breken didStreamed responses and used tool calls, but never both together on a billed account. The combination was planned and cut by the budget.

What changedThat combination is now tried first on every AI gateway. The miss stays in the record.

## 05What it means

- **Most escapes are invisible in code.** 71% need the product running with real data, users, failures or time.
- **They sit for months.** A median 152 days in the code before a user found them.
- **Running finds what reading cannot, with proof.** 61 of breken's 76 catches come with a reproduction that fails when run.
- **Head start is weeks, not hours.** A median 44 days before the user's report.

### Where review and breken fit

LayerSeesCatches

TestsWhat the developer thought ofKnown cases, every change

Code review, by people and AIThe diff, in contextLogic and design problems visible in code, before merge

brekenThe product running, with users, agents, data, outside failures and timeWhat only appears in production, before users

These bugs passed the first two layers by design: they were not in the diff. Review and breken are not alternatives; each covers what the other cannot see.

## 06Open benchmark

- The 327 bugs, their user reports, fixes and pinned commits, the harness, the judge prompts and every label are public.
- Any system can be submitted: one run record per product scan. We match, judge, check and publish it on the same page, with its agreement rate.
- A new version every quarter, mined from fixes merged after the newest model cutoff. Versions are never re-scored.

### Limits

- breken built the benchmark and ran every system. The prompt, the caps, the judge prompts and every label are public, and a vendor's own submission replaces our run of their system.
- Only bugs linked to a fix are included, so this under-counts what users hit.
- “What the bug needs” is labelled by people from the user report and the fix; borderline cases are listed in the dataset.
- 18 bugs ship no test, so a catch on them is judged and hand-confirmed, never confirmed by running.
- 31 bugs were reported before 1 July, so they were public when the models were trained. They are marked and not scored.
- Each result is a single run under the same spend cap; run-to-run variation is published with the data. Cost and wall time are recorded, not scored.
- No browser-agent products in this version. Only 2 bugs are multi-user.

**Corpus**sha256 ab1d3085ecb77c753932229c96e52ccc2a526fafbbb7beec75209cb98b8ce491  
**Judge prompts**sha256 f712b742f4cd2feabb6ec9e305293889666b62fec4cf277a689870c9bf364ab9  
**Frozen**v1 · 2026-10-19 11:28 UTC · 327 bugs · 33 products · 309 executable · 235 live at snapshot

[Try breken →](https://breken.ai/?signin=1)[Submit a system](mailto:support@breken.ai?subject=ESCAPES%20submission)

## 07Full methodology

Everything behind the numbers above, for anyone who wants to check or reproduce them.

How the dataset was built

MineIssues closed by a merged fix between 1 July and 19 October 2026.

VerifyEvery rule checked against GitHub: author, link, pre-fix commit, blame, tests.

Assemble5 to 12 bugs per product, both product kinds.*378 candidates*

ReviewA person reads the title, the user's words and the fix title of every candidate.*−51 removed*

Hand check20 bugs drawn at random, read against the issue, the fix and the pre-fix files.*19 pass · 1 caveat*

FreezeRules, products and bugs hashed. Every system and the scorer refuse a corpus that differs.*sha256 ab1d3085…*

#### The six rules

01**A user reported it.** Never a maintainer, member, bot, or someone who fixed their own bug.

02**The user could see it.** A wrong result, broken flow, lost data, wrong access or a crash. No docs, performance or build reports.

03**A merged fix is tied to it.** Linked by GitHub, same repository, changes real code. Fixes over 25 files are out.

04**Pinned at the fix's first parent.** Every fixed file is checked to exist there, at build time and at freeze time.

05**It was there when the user met it.** The fixed lines predate the report, by blame at that commit.

06**Tests make it executable.** 309 fixes ship a test, so a reproduction can fail before and pass after.

What is in the dataset

**Bugs by language**main language of the product

Go9 products

93

Python8 products

76

TypeScript7 products

63

PHP4 products

35

Ruby2 products

24

Java2 products

24

Rust1 product

12

**0****25****50****75****100 bugs**

KindProductsBugsProvable by runningLive at 1 JulyHigh severity

Web apps, APIs, MCP serverspeople and programs come to them2119017212836

SDKs, CLIs, toolkitsthey run inside other programs1213713710721

All3332730923557

**Live at 1 July:** the bug was already in the code at the snapshot and was reported after it, so a scan on 1 July could have caught it. These 235 are the bugs scored on this page.

#### By kind of failure

FailureBugsWhat the user sees

Broken flow139a job cannot be finished the way the product promises

SDK contract39wrong output or a failure for valid use of an SDK, CLI or API

Access and permission29a user sees or does what they should not, or cannot do what they should

Environment26appears only on a particular host, version, locale or configuration

Crash in normal use23an error or hang on an ordinary action

Lost input21something the user entered or uploaded is dropped

Wrong data17a number, record or result that is incorrect

Double action14one action applied twice: a charge, a message, a write

Outside-service failure9a partner, network or storage failure handled wrongly

Display8the screen shows the wrong thing for the state

Multi-user2two people acting at once

A bug can carry several classes; the table counts its main one. Classes are keyword guesses: a hand check of 20 corrected 4.

All 33 products

ProductWhat it isLanguageBugsLive at 1 July

Web apps, APIs, MCP servers · 21 products · 190 bugs

Arize-ai/phoenixLLM observability web appPython127

DependencyTrack/dependency-trackSupply-chain web app and APIJava1210

MHSanaei/3x-uiProxy admin panelGo128

QuantumNous/new-apiAI API gateway, web console and APIGo121

forem/foremCommunity platformRuby123

mudler/LocalAIOpenAI-compatible API serverGo1211

nautobot/nautobotNetwork source of truth, web and APIPython128

shopware/shopwareE-commerce platformPHP126

statamic/cmsCMSPHP1210

woodpecker-ci/woodpeckerCI server, web and APIGo1111

ether/etherpadCollaborative editorTypeScript109

OpenListTeam/OpenListFile list web appGo98

opensearch-project/OpenSearch-DashboardsSearch dashboardsTypeScript98

comet-ml/opikLLM evaluation web appPython64

modelcontextprotocol/serversReference MCP serversTypeScript66

openemr/openemrMedical records web appPHP65

Comfy-Org/ComfyUI\_frontendNode editor web appTypeScript54

SuiteCRM/SuiteCRM-CoreCRMPHP51

coder/coderRemote dev environments, web and APIGo53

cvat-ai/cvatAnnotation web appPython54

element-hq/synapseMatrix chat server, APIPython51

SDKs, CLIs, toolkits · 12 products · 137 bugs

Homebrew/brewPackage manager CLIRuby129

OpenAPITools/openapi-generatorCode generator CLIJava1210

PrefectHQ/fastmcpMCP SDKPython1211

gradio-app/gradioUI toolkitPython1211

pnpm/pnpmPackage manager CLIRust120

promptfoo/promptfooLLM eval CLITypeScript1210

pulumi/pulumiInfrastructure CLI and SDKsGo1210

rclone/rcloneStorage sync CLIGo1211

streamlit/streamlitApp toolkitPython1212

vercel/aiAI SDKTypeScript1212

resend/react-emailEmail toolkitTypeScript95

cli/cliGitHub CLIGo86

None of these products appears in any list breken develops, tunes or measures on. All 33 are now on the held-out lists our development guards read.

What a bug record holds

ReportIssue URL, a short quote of what the user saw, the date, the reporter's association. Never a handle.

FixThe pull request, its merge date, how it links to the issue, the files and lines it changed.

CommitsPre-fix (the fix's first parent) and fix. Systems run on the first; matches are proven against both.

ArrivalThe latest commit that touched the fixed lines before the report.

LabelsClass, direction, severity with a reason, executable, live at snapshot, public signal.

```
// one entry, shape only
{
  "id": "<owner>/<product>#<issue>",  "direction": "A",  "language": "Go",
  "issue": { "url": "…/issues/<n>", "createdAt": "<date>", "authorAssociation": "NONE",
             "userWords": "<what the user saw>", "signal": { "reactions": 0, "comments": 6 } },
  "fix": { "url": "…/pull/<n>", "mergedAt": "<date>", "howLinked": "closing reference" },
  "preFixSha": "<the fix's first parent>",  "fixSha": "<the fix commit>",
  "introducedAt": "<date>",
  "sites": [ { "path": "src/…/update.ts", "lines": [118, 131] } ],
  "tests": [ "src/…/update.test.ts" ],
  "snapshot": { "live": true },
  "class": ["broken-flow"],  "severity": "medium",  "executable": true
}
```

The systems and what they are given

brekenruns the productBoots the product at that commit into a world with users, data, outside services and time. Only bugs it would show a customer count.

Claude Fable 5.1reads the codeThrough Claude Code in a fresh session: the repository, its docs and a shell. Asked for the bugs real users will hit, at most 10 per product, with a reproduction where it can.

Claude Opus 5.5reads the codeThrough Claude Code, same session type, same prompt.

GPT-6 Astrareads the codeThrough Codex CLI, same prompt.

Static rulesSemgrepSemgrep's default rulesets, every hit. CodeQL or any SARIF tool can be imported.

Every system runs under the same spend cap per product. breken reports only what it would show a customer; the coding agents are asked for at most 10 findings per product; static rules return every hit. Code-review bots see a diff, not a product, so pointing one at the change that introduced a bug is an aimed run: published apart, never on the leaderboard.

#### What a system is given

A system is givenA repository and a commit, as a sealed workspace with its own history only, plus the docs inside it.

NeverA bug, an issue, a fix, a later commit, or any date after the commit.

It returnsFindings: a title, the failure a user would see, optional steps and a reproduction, and the files.

#### Two run modes

ModeRuns onScored onHead start

snapshotthe last commit before 1 July 2026, per productevery bug live there: 235snapshot to report

per-bugeach bug's pre-fix commitall 327the bug's whole window

This page shows snapshot runs: one scan on 1 July, measured against what users reported in the months after.

The prompt the agents get

Sent to Claude Code and Codex CLI as a fresh session: no memory, no project instructions, no tools beyond what the CLI ships. The product name and the commit are the only things filled in.

```
You are testing <owner/product> at the commit checked out in the current directory.
kind A: It is a product that people or programs use (a web app, an API or an MCP server).
kind B: It is an SDK, CLI or toolkit that people use to act on other things.

Find bugs that real users would run into in normal use: wrong results, broken flows,
lost data, wrong access, crashes, or (for a library or CLI) wrong output or failures
for valid use. You may read the code and docs, build it, run its tests and run it.
Report at most 10 bugs, best first. For each one, if you can, give a reproduction:
a shell command (and any new test file's full content) that FAILS on this commit
because of the bug and would pass once it is fixed.

Answer with JSON only:
{"findings":[{"id":"f1","title":"one sentence","failure":"what the user or caller sees go wrong",
  "steps":"how a user gets there",
  "reproduction":{"command":"...","files":{"path":"content"},"setup":"install command, if needed"},
  "files":["the source files at fault"]}]}
```

#### Finding contract

```
{
  "id": "f-012",
  "title": "Refund after a partial capture returns the full authorised amount",
  "failure": "The ledger refunds the authorised total, not the captured part",
  "steps": ["create an order", "capture part of it", "refund"],
  "reproduction": { "command": "npm test -- refunds.partial",
                    "files": { "test/refunds.partial.test.ts": "…" }, "setup": "npm ci" },
  "files": ["src/payments/refund.ts"],
  "location": { "path": "src/payments/refund.ts", "line": 88 }
}
```

Grading and metrics

A finding counts as a catch only if it names the same user-visible failure the user later reported. Three rungs decide it, in order. The higher rung wins.

01**By running it**A finding's reproduction is run in a container at the pre-fix commit and again at the fix commit. Fail, then pass: the catch is proven, and no judge is involved.

02**By the judge, for the rest**GPT-6 Sol, a model no system on the board runs on, reads the user's words, the fix's diff and the findings. It sees only what a finding says, never which system wrote it. Each finding is match, near or unrelated; unmatched findings are real, false or unsure against the code they cite. Prompts are hashed into the frozen rules.

03**By people**A person checks every judged match, near and false, plus a random 20% of the rest. The hand label wins. Agreement is published with every run.

#### Metrics

Escapes caughtbugs with ≥ 1 matching finding ÷ bugs scoredRecall. Proven and judged matches reported apart.

Confirmed by runningcatches whose reproduction fails at the pre-fix commit and passes at the fixThe rest are judged, then confirmed by a person.

False alarmsfindings matching no bug, labelled false against the code they cite“Real but unreported” and “unsure” are counted apart, never as false.

Precisionreal findings (a catch or an unreported defect) ÷ all findings

Weighted score(caught − false alarms) ÷ bugs scoredA catch earns one point, a false alarm loses one, everything else zero. As in RECON; shown beside recall, never instead of it. Static rules return every hit, uncapped; the weighted score punishes that.

#### What is not a match

—The same file or feature, but a different failure.

—A generic warning (“validation may be missing”, “a race is possible”) that does not name this failure.

—A failure the reporting user would not have seen.

Leakage controls

Held outNone of the 33 products appears in any list breken develops, tunes or measures on; all are now excluded from every lane.

SealedThe corpus lives outside the engine repository. A guard fails any engine source that names the sealed path.

FrozenEvery system, the matcher and the scorer refuse a corpus that differs from the pinned hash.

After cutoffOnly fixes merged from 1 July 2026, after the June 2026 cutoff of every model tested. 31 bugs were reported before 1 July: marked, and not scored in snapshot mode.

No answersThe workspace a system gets holds no history, files or tags from after its commit.

Commands

```
# build (read-only GitHub access; writes only to the sealed directory)
node bench/escapes/build.mjs --phase mine
node bench/escapes/build.mjs --phase verify
node bench/escapes/build.mjs --phase assemble --max-per-product 12 --min-per-product 5
node bench/escapes/freeze.mjs

# run a system
node bench/escapes/arms/run.mjs --arm breken    --label v1 --mode snapshot --cap-usd <usd>
node bench/escapes/arms/run.mjs --arm plain-llm --label v1 --mode snapshot --cap-usd <usd>
node bench/escapes/arms/run.mjs --arm static    --label v1 --mode snapshot

# match and score
node bench/escapes/match.mjs --label v1
node bench/escapes/score.mjs --label v1 --judge --model-env judge.env --cap-usd <usd>
node bench/escapes/score.mjs --label v1
```

[breken▚▚](https://breken.ai/)ESCAPES v1 · October 2026 · dataset frozen 19 October 2026
