{"article":{"slug":"escapes-a-benchmark-of-327-bugs-that-reached-real-users","title":"ESCAPES: a benchmark of 327 bugs that reached real users","subtitle":null,"summary":"breken's open ESCAPES benchmark rewinds 33 open-source products to 1 July 2026 code and scores 235 user-reported bugs: breken's run-based system caught 76 (32%) with 20 false alarms versus 37 for Claude Fable 5.1 in Claude Code; vendor-run results with public data and judge prompts.","content_type":"research","language":"en","canonical_url":"https://breken.ai/benchmark","author":{"name":null,"url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"breken","url":"https://breken.ai/","listing_slug":"breken","listing":{"slug":"breken","name":"breken","listing_type":"company","url":"https://listedstartups.com/companies/breken"}},"topics":[{"name":"benchmarks","slug":"benchmarks","url":"https://listedarticles.com/topics/benchmarks"},{"name":"software-engineering","slug":"software-engineering","url":"https://listedarticles.com/topics/software-engineering"},{"name":"ai-agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"developer-tools","slug":"developer-tools","url":"https://listedarticles.com/topics/developer-tools"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":3592,"reading_minutes":16,"published_at":"2026-11-14T11:13:44.585Z","added_at":"2026-11-14T11:13:44.585Z","updated_at":"2026-11-14T11:13:44.585Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/escapes-a-benchmark-of-327-bugs-that-reached-real-users","markdown_url":"https://listedarticles.com/articles/escapes-a-benchmark-of-327-bugs-that-reached-real-users.md","example":false,"citation":"breken. \"ESCAPES: a benchmark of 327 bugs that reached real users.\" 14 Nov 2026. https://breken.ai/benchmark (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://breken.ai/benchmark"},"body_markdown":"Benchmark·October 2026\n\n# ESCAPES\n\n327 bugs that passed code review and tests, reached real users, and were fixed. We rewound 33 open-source products to the code before each fix and asked one question: how many could have been caught before a user hit them?\n\nEscapes caught before the user reported them\n\nOne scan per product, on its code as of 1 July 2026. 235 of the 327 bugs were already in that code and were reported by users in the weeks after. The share of those 235 each system had already found.\n\nbrekenruns the product\n\n76 bugs · 32%\n\nClaude Fable 5.1Claude Code\n\n37 bugs · 16%\n\nClaude Opus 5.5Claude Code\n\n33 bugs · 14%\n\nGPT-6 AstraCodex CLI\n\n28 bugs · 12%\n\nStatic rulesSemgrep\n\n9 bugs · 4%\n\n**0****10****20****30****40% of 235**\n\nbreken found 76 of the 235, with 20 false alarms across the 33 products. The best model reading the code found 37, with 59 false alarms. Full table in section 03.\n\nContents\n\n1. [01What escapes](#escapes)\n2. [02How it was measured](#measured)\n3. [03Catching them first](#results)\n4. [04Four escapes](#cases)\n5. [05What it means](#means)\n6. [06Open benchmark](#open)\n7. [07Full methodology](#method)\n\n## 01What escapes\n\nThe bugs\n\n**327**real, user-reported bugs from 33 open-source products\n\n**152 days**median time a bug sat in the code before a user reported it\n\n**53 days**median gap between the 1 July snapshot and the user's report\n\n**71%**only appear when the product runs: data, users, failures or time\n\nEvery one of these bugs got through a project's own review and test suite. That is not a failure of review: most of them cannot be seen in a diff. They need a real data shape, a second user, a partner's outage or a month to pass. That is the layer ESCAPES measures.\n\n**Caught, by what the bug needs**share of each row's bugs · 235 in all\n\nEach bug is labelled by what it takes to see it. Per row: the share caught by the best model reading the code, Claude Fable 5.1 through Claude Code, and by breken running the product.\n\nClaude Fable 5.1 reading the code (Claude Code)breken, running the product\n\nVisible by reading one file69\n\n*33%*\n\n*32%*\n\nNeeds the product running51\n\n*12%*\n\n*39%*\n\nNeeds a real data shape42\n\n*7%*\n\n*33%*\n\nNeeds a second user or concurrency22\n\n*5%*\n\n*32%*\n\nNeeds an outside service to fail19\n\n*5%*\n\n*37%*\n\nNeeds time to pass (renewal, expiry, upgrade)17\n\n*6%*\n\n*24%*\n\nNeeds a specific environment15\n\n*13%*\n\n*13%*\n\n**0****10****20****30****40%**\n\nReading the code and running the product find about the same share of bugs visible in one file. Beyond that, Claude Fable 5.1 finds 14 of 166; running finds 54.\n\n## 02How it was measured\n\n- **The bugs:** reported by a real user on the product's public tracker, fixed by a merged pull request between 1 July and 19 October 2026 (after every tested model's training cutoff), and present in the code when the user hit them.\n- **The run:** the product's code and docs as of 1 July 2026, nothing later. The system reports the bugs it expects users to hit.\n- **Why 1 July:** the first day after the training cutoff of every model tested, so none of them had seen these bugs or their fixes. One scan on that day is what a customer would get. 235 of the 327 bugs were already in the code by then; those are the ones scored. The other 92 arrived later.\n- **A catch:** the finding describes the failure the user later reported. Wherever possible it is confirmed by running: its reproduction fails before the fix and passes after.\n- **Checking:** an independent judge model, one that no system on the board runs on, grades the rest. People check every match, every near miss and a 20% sample.\n- **Reference lines:** frontier AI models reading the code through their own coding agents (Claude Code, Codex CLI), with the same budget, to show what reading alone finds. Static rules (Semgrep) as the floor.\n- **The score:** escapes caught, as a share of the 235. Beside it a weighted score, as in RECON: a catch earns one point, a false alarm loses one.\n\nThe full dataset rules, the exact prompt the agents get, the judge and every metric definition are in section 07.\n\n## 03Catching them first\n\nbreken caught 76 of the 235 escapes, 61 of them confirmed by running. The 3 frontier models reading the code caught 28 to 37; static rules caught 9.\n\n**Figure 1 · Weighted score by system**(caught − false alarms) ÷ 235\n\nA catch earns one point and a false alarm loses one, divided by the 235 bugs. Below zero, a system raised more false alarms than bugs it caught.\n\nbreken · runs the productcaught 76 · false alarms 20\n\n+23.8%\n\nClaude Fable 5.1 · Claude Codecaught 37 · false alarms 59\n\n−9.4%\n\nClaude Opus 5.5 · Claude Codecaught 33 · false alarms 69\n\n−15.3%\n\nGPT-6 Astra · Codex CLIcaught 28 · false alarms 86\n\n−24.7%\n\nStatic rules · Semgrepcaught 9 · false alarms 112\n\n−43.8%\n\n**−50%****−25%****0****+25%****+50%**\n\n**Table 1 · Full results**one scan per product · one run per system\n\nSystemWeightedCaughtBeyond 1 fileProvenFalse alarmsUnreportedPrecisionHead start\n\n1brekenruns the product+23.8%76 · 32%54 of 1666120 · 0.63885%44 days\n\n2Claude Fable 5.1Claude Code−9.4%37 · 16%14 of 166959 · 1.81346%39 days\n\n3Claude Opus 5.5Claude Code−15.3%33 · 14%12 of 166769 · 2.11139%37 days\n\n4GPT-6 AstraCodex CLI−24.7%28 · 12%10 of 166586 · 2.6930%34 days\n\n5Static rulesSemgrep−43.8%9 · 4%0 of 1660112 · 3.4612%26 days\n\n**Caught:** escapes caught, of the 235. **Beyond 1 file:** caught among the 166 that need the product running. **Proven:** catches with a reproduction that fails before the fix and passes after; the rest judged, then confirmed by a person. **False alarms:** total, then per product. **Unreported:** real defects no user had reported yet. **Precision:** real findings ÷ all findings. **Head start:** median days between the scan and the user's report, over caught bugs.\n\n## 04Four escapes\n\nThree catches and one miss.\n\n### An e-commerce platform: a refund after a partial captureCaught 47 days before the user\n\nWeb app · needs a real data shape · proven by running\n\nThe userRefunding an order after a partial capture refunded the full authorised amount.\n\nWhat it neededA partial-capture state. The change that introduced the bug never touched it.\n\nWhat breken didRan checkout with a partial capture, then refunded. The ledger showed the full amount; the same steps without the partial capture were correct.\n\nProof**Proven by running.** Its reproduction failed before the fix and passed after.\n\n### A storage sync CLI: a connection dropped mid-transferCaught 63 days before the user\n\nCLI · needs an outside service to fail · proven by running\n\nThe userA transfer cut off by a dropped connection was reported as complete. The half-copied file stayed, marked as synced.\n\nWhat it neededA network drop at the wrong moment, then a second sync.\n\nWhat breken didCut the connection mid-transfer, synced again, compared checksums on both sides. They differed.\n\nProof**Proven by running.** Two commands reproduce it.\n\n### A collaborative editor: two editors and a reconnectCaught 29 days before the user\n\nWeb app · needs a second user and a failure · judged, confirmed by a person\n\nThe userAfter a short disconnect, one person's edits overwrote another's.\n\nWhat it neededTwo people on one paragraph and a dropped connection at the same moment.\n\nWhat breken didTwo simulated editors worked on one paragraph while one connection dropped and came back. An edit was lost in 3 of 3 runs, never without the drop.\n\nProof**Judged, then confirmed.** Two live sessions cannot run in the matcher; the judge said match and a person agreed.\n\n### A miss: billing for streamed tool calls in an AI gatewayNot caught\n\nAPI gateway · needs the product running · missed\n\nThe userStreamed responses with tool calls were billed twice for the tool output.\n\nWhat it neededStreaming and tool calls together, on a billed account.\n\nWhat breken didStreamed responses and used tool calls, but never both together on a billed account. The combination was planned and cut by the budget.\n\nWhat changedThat combination is now tried first on every AI gateway. The miss stays in the record.\n\n## 05What it means\n\n- **Most escapes are invisible in code.** 71% need the product running with real data, users, failures or time.\n- **They sit for months.** A median 152 days in the code before a user found them.\n- **Running finds what reading cannot, with proof.** 61 of breken's 76 catches come with a reproduction that fails when run.\n- **Head start is weeks, not hours.** A median 44 days before the user's report.\n\n### Where review and breken fit\n\nLayerSeesCatches\n\nTestsWhat the developer thought ofKnown cases, every change\n\nCode review, by people and AIThe diff, in contextLogic and design problems visible in code, before merge\n\nbrekenThe product running, with users, agents, data, outside failures and timeWhat only appears in production, before users\n\nThese bugs passed the first two layers by design: they were not in the diff. Review and breken are not alternatives; each covers what the other cannot see.\n\n## 06Open benchmark\n\n- The 327 bugs, their user reports, fixes and pinned commits, the harness, the judge prompts and every label are public.\n- Any system can be submitted: one run record per product scan. We match, judge, check and publish it on the same page, with its agreement rate.\n- A new version every quarter, mined from fixes merged after the newest model cutoff. Versions are never re-scored.\n\n### Limits\n\n- breken built the benchmark and ran every system. The prompt, the caps, the judge prompts and every label are public, and a vendor's own submission replaces our run of their system.\n- Only bugs linked to a fix are included, so this under-counts what users hit.\n- “What the bug needs” is labelled by people from the user report and the fix; borderline cases are listed in the dataset.\n- 18 bugs ship no test, so a catch on them is judged and hand-confirmed, never confirmed by running.\n- 31 bugs were reported before 1 July, so they were public when the models were trained. They are marked and not scored.\n- Each result is a single run under the same spend cap; run-to-run variation is published with the data. Cost and wall time are recorded, not scored.\n- No browser-agent products in this version. Only 2 bugs are multi-user.\n\n**Corpus**sha256 ab1d3085ecb77c753932229c96e52ccc2a526fafbbb7beec75209cb98b8ce491  \n**Judge prompts**sha256 f712b742f4cd2feabb6ec9e305293889666b62fec4cf277a689870c9bf364ab9  \n**Frozen**v1 · 2026-10-19 11:28 UTC · 327 bugs · 33 products · 309 executable · 235 live at snapshot\n\n[Try breken →](https://breken.ai/?signin=1)[Submit a system](mailto:support@breken.ai?subject=ESCAPES%20submission)\n\n## 07Full methodology\n\nEverything behind the numbers above, for anyone who wants to check or reproduce them.\n\nHow the dataset was built\n\nMineIssues closed by a merged fix between 1 July and 19 October 2026.\n\nVerifyEvery rule checked against GitHub: author, link, pre-fix commit, blame, tests.\n\nAssemble5 to 12 bugs per product, both product kinds.*378 candidates*\n\nReviewA person reads the title, the user's words and the fix title of every candidate.*−51 removed*\n\nHand check20 bugs drawn at random, read against the issue, the fix and the pre-fix files.*19 pass · 1 caveat*\n\nFreezeRules, products and bugs hashed. Every system and the scorer refuse a corpus that differs.*sha256 ab1d3085…*\n\n#### The six rules\n\n01**A user reported it.** Never a maintainer, member, bot, or someone who fixed their own bug.\n\n02**The user could see it.** A wrong result, broken flow, lost data, wrong access or a crash. No docs, performance or build reports.\n\n03**A merged fix is tied to it.** Linked by GitHub, same repository, changes real code. Fixes over 25 files are out.\n\n04**Pinned at the fix's first parent.** Every fixed file is checked to exist there, at build time and at freeze time.\n\n05**It was there when the user met it.** The fixed lines predate the report, by blame at that commit.\n\n06**Tests make it executable.** 309 fixes ship a test, so a reproduction can fail before and pass after.\n\nWhat is in the dataset\n\n**Bugs by language**main language of the product\n\nGo9 products\n\n93\n\nPython8 products\n\n76\n\nTypeScript7 products\n\n63\n\nPHP4 products\n\n35\n\nRuby2 products\n\n24\n\nJava2 products\n\n24\n\nRust1 product\n\n12\n\n**0****25****50****75****100 bugs**\n\nKindProductsBugsProvable by runningLive at 1 JulyHigh severity\n\nWeb apps, APIs, MCP serverspeople and programs come to them2119017212836\n\nSDKs, CLIs, toolkitsthey run inside other programs1213713710721\n\nAll3332730923557\n\n**Live at 1 July:** the bug was already in the code at the snapshot and was reported after it, so a scan on 1 July could have caught it. These 235 are the bugs scored on this page.\n\n#### By kind of failure\n\nFailureBugsWhat the user sees\n\nBroken flow139a job cannot be finished the way the product promises\n\nSDK contract39wrong output or a failure for valid use of an SDK, CLI or API\n\nAccess and permission29a user sees or does what they should not, or cannot do what they should\n\nEnvironment26appears only on a particular host, version, locale or configuration\n\nCrash in normal use23an error or hang on an ordinary action\n\nLost input21something the user entered or uploaded is dropped\n\nWrong data17a number, record or result that is incorrect\n\nDouble action14one action applied twice: a charge, a message, a write\n\nOutside-service failure9a partner, network or storage failure handled wrongly\n\nDisplay8the screen shows the wrong thing for the state\n\nMulti-user2two people acting at once\n\nA bug can carry several classes; the table counts its main one. Classes are keyword guesses: a hand check of 20 corrected 4.\n\nAll 33 products\n\nProductWhat it isLanguageBugsLive at 1 July\n\nWeb apps, APIs, MCP servers · 21 products · 190 bugs\n\nArize-ai/phoenixLLM observability web appPython127\n\nDependencyTrack/dependency-trackSupply-chain web app and APIJava1210\n\nMHSanaei/3x-uiProxy admin panelGo128\n\nQuantumNous/new-apiAI API gateway, web console and APIGo121\n\nforem/foremCommunity platformRuby123\n\nmudler/LocalAIOpenAI-compatible API serverGo1211\n\nnautobot/nautobotNetwork source of truth, web and APIPython128\n\nshopware/shopwareE-commerce platformPHP126\n\nstatamic/cmsCMSPHP1210\n\nwoodpecker-ci/woodpeckerCI server, web and APIGo1111\n\nether/etherpadCollaborative editorTypeScript109\n\nOpenListTeam/OpenListFile list web appGo98\n\nopensearch-project/OpenSearch-DashboardsSearch dashboardsTypeScript98\n\ncomet-ml/opikLLM evaluation web appPython64\n\nmodelcontextprotocol/serversReference MCP serversTypeScript66\n\nopenemr/openemrMedical records web appPHP65\n\nComfy-Org/ComfyUI\\_frontendNode editor web appTypeScript54\n\nSuiteCRM/SuiteCRM-CoreCRMPHP51\n\ncoder/coderRemote dev environments, web and APIGo53\n\ncvat-ai/cvatAnnotation web appPython54\n\nelement-hq/synapseMatrix chat server, APIPython51\n\nSDKs, CLIs, toolkits · 12 products · 137 bugs\n\nHomebrew/brewPackage manager CLIRuby129\n\nOpenAPITools/openapi-generatorCode generator CLIJava1210\n\nPrefectHQ/fastmcpMCP SDKPython1211\n\ngradio-app/gradioUI toolkitPython1211\n\npnpm/pnpmPackage manager CLIRust120\n\npromptfoo/promptfooLLM eval CLITypeScript1210\n\npulumi/pulumiInfrastructure CLI and SDKsGo1210\n\nrclone/rcloneStorage sync CLIGo1211\n\nstreamlit/streamlitApp toolkitPython1212\n\nvercel/aiAI SDKTypeScript1212\n\nresend/react-emailEmail toolkitTypeScript95\n\ncli/cliGitHub CLIGo86\n\nNone of these products appears in any list breken develops, tunes or measures on. All 33 are now on the held-out lists our development guards read.\n\nWhat a bug record holds\n\nReportIssue URL, a short quote of what the user saw, the date, the reporter's association. Never a handle.\n\nFixThe pull request, its merge date, how it links to the issue, the files and lines it changed.\n\nCommitsPre-fix (the fix's first parent) and fix. Systems run on the first; matches are proven against both.\n\nArrivalThe latest commit that touched the fixed lines before the report.\n\nLabelsClass, direction, severity with a reason, executable, live at snapshot, public signal.\n\n```\n// one entry, shape only\n{\n  \"id\": \"<owner>/<product>#<issue>\",  \"direction\": \"A\",  \"language\": \"Go\",\n  \"issue\": { \"url\": \"…/issues/<n>\", \"createdAt\": \"<date>\", \"authorAssociation\": \"NONE\",\n             \"userWords\": \"<what the user saw>\", \"signal\": { \"reactions\": 0, \"comments\": 6 } },\n  \"fix\": { \"url\": \"…/pull/<n>\", \"mergedAt\": \"<date>\", \"howLinked\": \"closing reference\" },\n  \"preFixSha\": \"<the fix's first parent>\",  \"fixSha\": \"<the fix commit>\",\n  \"introducedAt\": \"<date>\",\n  \"sites\": [ { \"path\": \"src/…/update.ts\", \"lines\": [118, 131] } ],\n  \"tests\": [ \"src/…/update.test.ts\" ],\n  \"snapshot\": { \"live\": true },\n  \"class\": [\"broken-flow\"],  \"severity\": \"medium\",  \"executable\": true\n}\n```\n\nThe systems and what they are given\n\nbrekenruns the productBoots the product at that commit into a world with users, data, outside services and time. Only bugs it would show a customer count.\n\nClaude Fable 5.1reads the codeThrough Claude Code in a fresh session: the repository, its docs and a shell. Asked for the bugs real users will hit, at most 10 per product, with a reproduction where it can.\n\nClaude Opus 5.5reads the codeThrough Claude Code, same session type, same prompt.\n\nGPT-6 Astrareads the codeThrough Codex CLI, same prompt.\n\nStatic rulesSemgrepSemgrep's default rulesets, every hit. CodeQL or any SARIF tool can be imported.\n\nEvery system runs under the same spend cap per product. breken reports only what it would show a customer; the coding agents are asked for at most 10 findings per product; static rules return every hit. Code-review bots see a diff, not a product, so pointing one at the change that introduced a bug is an aimed run: published apart, never on the leaderboard.\n\n#### What a system is given\n\nA system is givenA repository and a commit, as a sealed workspace with its own history only, plus the docs inside it.\n\nNeverA bug, an issue, a fix, a later commit, or any date after the commit.\n\nIt returnsFindings: a title, the failure a user would see, optional steps and a reproduction, and the files.\n\n#### Two run modes\n\nModeRuns onScored onHead start\n\nsnapshotthe last commit before 1 July 2026, per productevery bug live there: 235snapshot to report\n\nper-bugeach bug's pre-fix commitall 327the bug's whole window\n\nThis page shows snapshot runs: one scan on 1 July, measured against what users reported in the months after.\n\nThe prompt the agents get\n\nSent to Claude Code and Codex CLI as a fresh session: no memory, no project instructions, no tools beyond what the CLI ships. The product name and the commit are the only things filled in.\n\n```\nYou are testing <owner/product> at the commit checked out in the current directory.\nkind A: It is a product that people or programs use (a web app, an API or an MCP server).\nkind B: It is an SDK, CLI or toolkit that people use to act on other things.\n\nFind bugs that real users would run into in normal use: wrong results, broken flows,\nlost data, wrong access, crashes, or (for a library or CLI) wrong output or failures\nfor valid use. You may read the code and docs, build it, run its tests and run it.\nReport at most 10 bugs, best first. For each one, if you can, give a reproduction:\na shell command (and any new test file's full content) that FAILS on this commit\nbecause of the bug and would pass once it is fixed.\n\nAnswer with JSON only:\n{\"findings\":[{\"id\":\"f1\",\"title\":\"one sentence\",\"failure\":\"what the user or caller sees go wrong\",\n  \"steps\":\"how a user gets there\",\n  \"reproduction\":{\"command\":\"...\",\"files\":{\"path\":\"content\"},\"setup\":\"install command, if needed\"},\n  \"files\":[\"the source files at fault\"]}]}\n```\n\n#### Finding contract\n\n```\n{\n  \"id\": \"f-012\",\n  \"title\": \"Refund after a partial capture returns the full authorised amount\",\n  \"failure\": \"The ledger refunds the authorised total, not the captured part\",\n  \"steps\": [\"create an order\", \"capture part of it\", \"refund\"],\n  \"reproduction\": { \"command\": \"npm test -- refunds.partial\",\n                    \"files\": { \"test/refunds.partial.test.ts\": \"…\" }, \"setup\": \"npm ci\" },\n  \"files\": [\"src/payments/refund.ts\"],\n  \"location\": { \"path\": \"src/payments/refund.ts\", \"line\": 88 }\n}\n```\n\nGrading and metrics\n\nA finding counts as a catch only if it names the same user-visible failure the user later reported. Three rungs decide it, in order. The higher rung wins.\n\n01**By running it**A finding's reproduction is run in a container at the pre-fix commit and again at the fix commit. Fail, then pass: the catch is proven, and no judge is involved.\n\n02**By the judge, for the rest**GPT-6 Sol, a model no system on the board runs on, reads the user's words, the fix's diff and the findings. It sees only what a finding says, never which system wrote it. Each finding is match, near or unrelated; unmatched findings are real, false or unsure against the code they cite. Prompts are hashed into the frozen rules.\n\n03**By people**A person checks every judged match, near and false, plus a random 20% of the rest. The hand label wins. Agreement is published with every run.\n\n#### Metrics\n\nEscapes caughtbugs with ≥ 1 matching finding ÷ bugs scoredRecall. Proven and judged matches reported apart.\n\nConfirmed by runningcatches whose reproduction fails at the pre-fix commit and passes at the fixThe rest are judged, then confirmed by a person.\n\nFalse alarmsfindings matching no bug, labelled false against the code they cite“Real but unreported” and “unsure” are counted apart, never as false.\n\nPrecisionreal findings (a catch or an unreported defect) ÷ all findings\n\nWeighted score(caught − false alarms) ÷ bugs scoredA catch earns one point, a false alarm loses one, everything else zero. As in RECON; shown beside recall, never instead of it. Static rules return every hit, uncapped; the weighted score punishes that.\n\n#### What is not a match\n\n—The same file or feature, but a different failure.\n\n—A generic warning (“validation may be missing”, “a race is possible”) that does not name this failure.\n\n—A failure the reporting user would not have seen.\n\nLeakage controls\n\nHeld outNone of the 33 products appears in any list breken develops, tunes or measures on; all are now excluded from every lane.\n\nSealedThe corpus lives outside the engine repository. A guard fails any engine source that names the sealed path.\n\nFrozenEvery system, the matcher and the scorer refuse a corpus that differs from the pinned hash.\n\nAfter cutoffOnly fixes merged from 1 July 2026, after the June 2026 cutoff of every model tested. 31 bugs were reported before 1 July: marked, and not scored in snapshot mode.\n\nNo answersThe workspace a system gets holds no history, files or tags from after its commit.\n\nCommands\n\n```\n# build (read-only GitHub access; writes only to the sealed directory)\nnode bench/escapes/build.mjs --phase mine\nnode bench/escapes/build.mjs --phase verify\nnode bench/escapes/build.mjs --phase assemble --max-per-product 12 --min-per-product 5\nnode bench/escapes/freeze.mjs\n\n# run a system\nnode bench/escapes/arms/run.mjs --arm breken    --label v1 --mode snapshot --cap-usd <usd>\nnode bench/escapes/arms/run.mjs --arm plain-llm --label v1 --mode snapshot --cap-usd <usd>\nnode bench/escapes/arms/run.mjs --arm static    --label v1 --mode snapshot\n\n# match and score\nnode bench/escapes/match.mjs --label v1\nnode bench/escapes/score.mjs --label v1 --judge --model-env judge.env --cap-usd <usd>\nnode bench/escapes/score.mjs --label v1\n```\n\n[breken▚▚](https://breken.ai/)ESCAPES v1 · October 2026 · dataset frozen 19 October 2026\n","body_html":"<p>Benchmark·October 2026</p>\n<h1 id=\"escapes\">ESCAPES</h1>\n<p>327 bugs that passed code review and tests, reached real users, and were fixed. We rewound 33 open-source products to the code before each fix and asked one question: how many could have been caught before a user hit them?</p>\n<p>Escapes caught before the user reported them</p>\n<p>One scan per product, on its code as of 1 July 2026. 235 of the 327 bugs were already in that code and were reported by users in the weeks after. The share of those 235 each system had already found.</p>\n<p>brekenruns the product</p>\n<p>76 bugs · 32%</p>\n<p>Claude Fable 5.1Claude Code</p>\n<p>37 bugs · 16%</p>\n<p>Claude Opus 5.5Claude Code</p>\n<p>33 bugs · 14%</p>\n<p>GPT-6 AstraCodex CLI</p>\n<p>28 bugs · 12%</p>\n<p>Static rulesSemgrep</p>\n<p>9 bugs · 4%</p>\n<p><strong>0<strong><em><em>10</em></strong></em>20<strong><em><em>30</em></strong></em>40% of 235</strong></p>\n<p>breken found 76 of the 235, with 20 false alarms across the 33 products. The best model reading the code found 37, with 59 false alarms. Full table in section 03.</p>\n<p>Contents</p>\n<ol><li><a href=\"#escapes\">01What escapes</a></li><li><a href=\"#measured\">02How it was measured</a></li><li><a href=\"#results\">03Catching them first</a></li><li><a href=\"#cases\">04Four escapes</a></li><li><a href=\"#means\">05What it means</a></li><li><a href=\"#open\">06Open benchmark</a></li><li><a href=\"#method\">07Full methodology</a></li></ol>\n<h2 id=\"01what-escapes\">01What escapes</h2>\n<p>The bugs</p>\n<p><strong>327</strong>real, user-reported bugs from 33 open-source products</p>\n<p><strong>152 days</strong>median time a bug sat in the code before a user reported it</p>\n<p><strong>53 days</strong>median gap between the 1 July snapshot and the user&#39;s report</p>\n<p><strong>71%</strong>only appear when the product runs: data, users, failures or time</p>\n<p>Every one of these bugs got through a project&#39;s own review and test suite. That is not a failure of review: most of them cannot be seen in a diff. They need a real data shape, a second user, a partner&#39;s outage or a month to pass. That is the layer ESCAPES measures.</p>\n<p><strong>Caught, by what the bug needs</strong>share of each row&#39;s bugs · 235 in all</p>\n<p>Each bug is labelled by what it takes to see it. Per row: the share caught by the best model reading the code, Claude Fable 5.1 through Claude Code, and by breken running the product.</p>\n<p>Claude Fable 5.1 reading the code (Claude Code)breken, running the product</p>\n<p>Visible by reading one file69</p>\n<p><em>33%</em></p>\n<p><em>32%</em></p>\n<p>Needs the product running51</p>\n<p><em>12%</em></p>\n<p><em>39%</em></p>\n<p>Needs a real data shape42</p>\n<p><em>7%</em></p>\n<p><em>33%</em></p>\n<p>Needs a second user or concurrency22</p>\n<p><em>5%</em></p>\n<p><em>32%</em></p>\n<p>Needs an outside service to fail19</p>\n<p><em>5%</em></p>\n<p><em>37%</em></p>\n<p>Needs time to pass (renewal, expiry, upgrade)17</p>\n<p><em>6%</em></p>\n<p><em>24%</em></p>\n<p>Needs a specific environment15</p>\n<p><em>13%</em></p>\n<p><em>13%</em></p>\n<p><strong>0<strong><em><em>10</em></strong></em>20<strong><em><em>30</em></strong></em>40%</strong></p>\n<p>Reading the code and running the product find about the same share of bugs visible in one file. Beyond that, Claude Fable 5.1 finds 14 of 166; running finds 54.</p>\n<h2 id=\"02how-it-was-measured\">02How it was measured</h2>\n<ul><li><strong>The bugs:</strong> reported by a real user on the product&#39;s public tracker, fixed by a merged pull request between 1 July and 19 October 2026 (after every tested model&#39;s training cutoff), and present in the code when the user hit them.</li><li><strong>The run:</strong> the product&#39;s code and docs as of 1 July 2026, nothing later. The system reports the bugs it expects users to hit.</li><li><strong>Why 1 July:</strong> the first day after the training cutoff of every model tested, so none of them had seen these bugs or their fixes. One scan on that day is what a customer would get. 235 of the 327 bugs were already in the code by then; those are the ones scored. The other 92 arrived later.</li><li><strong>A catch:</strong> the finding describes the failure the user later reported. Wherever possible it is confirmed by running: its reproduction fails before the fix and passes after.</li><li><strong>Checking:</strong> an independent judge model, one that no system on the board runs on, grades the rest. People check every match, every near miss and a 20% sample.</li><li><strong>Reference lines:</strong> frontier AI models reading the code through their own coding agents (Claude Code, Codex CLI), with the same budget, to show what reading alone finds. Static rules (Semgrep) as the floor.</li><li><strong>The score:</strong> escapes caught, as a share of the 235. Beside it a weighted score, as in RECON: a catch earns one point, a false alarm loses one.</li></ul>\n<p>The full dataset rules, the exact prompt the agents get, the judge and every metric definition are in section 07.</p>\n<h2 id=\"03catching-them-first\">03Catching them first</h2>\n<p>breken caught 76 of the 235 escapes, 61 of them confirmed by running. The 3 frontier models reading the code caught 28 to 37; static rules caught 9.</p>\n<p><strong>Figure 1 · Weighted score by system</strong>(caught − false alarms) ÷ 235</p>\n<p>A catch earns one point and a false alarm loses one, divided by the 235 bugs. Below zero, a system raised more false alarms than bugs it caught.</p>\n<p>breken · runs the productcaught 76 · false alarms 20</p>\n<p>+23.8%</p>\n<p>Claude Fable 5.1 · Claude Codecaught 37 · false alarms 59</p>\n<p>−9.4%</p>\n<p>Claude Opus 5.5 · Claude Codecaught 33 · false alarms 69</p>\n<p>−15.3%</p>\n<p>GPT-6 Astra · Codex CLIcaught 28 · false alarms 86</p>\n<p>−24.7%</p>\n<p>Static rules · Semgrepcaught 9 · false alarms 112</p>\n<p>−43.8%</p>\n<p><strong>−50%<strong><em><em>−25%</em></strong></em>0<strong><em><em>+25%</em></strong></em>+50%</strong></p>\n<p><strong>Table 1 · Full results</strong>one scan per product · one run per system</p>\n<p>SystemWeightedCaughtBeyond 1 fileProvenFalse alarmsUnreportedPrecisionHead start</p>\n<p>1brekenruns the product+23.8%76 · 32%54 of 1666120 · 0.63885%44 days</p>\n<p>2Claude Fable 5.1Claude Code−9.4%37 · 16%14 of 166959 · 1.81346%39 days</p>\n<p>3Claude Opus 5.5Claude Code−15.3%33 · 14%12 of 166769 · 2.11139%37 days</p>\n<p>4GPT-6 AstraCodex CLI−24.7%28 · 12%10 of 166586 · 2.6930%34 days</p>\n<p>5Static rulesSemgrep−43.8%9 · 4%0 of 1660112 · 3.4612%26 days</p>\n<p><strong>Caught:</strong> escapes caught, of the 235. <strong>Beyond 1 file:</strong> caught among the 166 that need the product running. <strong>Proven:</strong> catches with a reproduction that fails before the fix and passes after; the rest judged, then confirmed by a person. <strong>False alarms:</strong> total, then per product. <strong>Unreported:</strong> real defects no user had reported yet. <strong>Precision:</strong> real findings ÷ all findings. <strong>Head start:</strong> median days between the scan and the user&#39;s report, over caught bugs.</p>\n<h2 id=\"04four-escapes\">04Four escapes</h2>\n<p>Three catches and one miss.</p>\n<h3 id=\"an-e-commerce-platform-a-refund-after-a-partial-capturecaught-47\">An e-commerce platform: a refund after a partial captureCaught 47 days before the user</h3>\n<p>Web app · needs a real data shape · proven by running</p>\n<p>The userRefunding an order after a partial capture refunded the full authorised amount.</p>\n<p>What it neededA partial-capture state. The change that introduced the bug never touched it.</p>\n<p>What breken didRan checkout with a partial capture, then refunded. The ledger showed the full amount; the same steps without the partial capture were correct.</p>\n<p>Proof<strong>Proven by running.</strong> Its reproduction failed before the fix and passed after.</p>\n<h3 id=\"a-storage-sync-cli-a-connection-dropped-mid-transfercaught-63-da\">A storage sync CLI: a connection dropped mid-transferCaught 63 days before the user</h3>\n<p>CLI · needs an outside service to fail · proven by running</p>\n<p>The userA transfer cut off by a dropped connection was reported as complete. The half-copied file stayed, marked as synced.</p>\n<p>What it neededA network drop at the wrong moment, then a second sync.</p>\n<p>What breken didCut the connection mid-transfer, synced again, compared checksums on both sides. They differed.</p>\n<p>Proof<strong>Proven by running.</strong> Two commands reproduce it.</p>\n<h3 id=\"a-collaborative-editor-two-editors-and-a-reconnectcaught-29-days\">A collaborative editor: two editors and a reconnectCaught 29 days before the user</h3>\n<p>Web app · needs a second user and a failure · judged, confirmed by a person</p>\n<p>The userAfter a short disconnect, one person&#39;s edits overwrote another&#39;s.</p>\n<p>What it neededTwo people on one paragraph and a dropped connection at the same moment.</p>\n<p>What breken didTwo simulated editors worked on one paragraph while one connection dropped and came back. An edit was lost in 3 of 3 runs, never without the drop.</p>\n<p>Proof<strong>Judged, then confirmed.</strong> Two live sessions cannot run in the matcher; the judge said match and a person agreed.</p>\n<h3 id=\"a-miss-billing-for-streamed-tool-calls-in-an-ai-gatewaynot-caugh\">A miss: billing for streamed tool calls in an AI gatewayNot caught</h3>\n<p>API gateway · needs the product running · missed</p>\n<p>The userStreamed responses with tool calls were billed twice for the tool output.</p>\n<p>What it neededStreaming and tool calls together, on a billed account.</p>\n<p>What breken didStreamed responses and used tool calls, but never both together on a billed account. The combination was planned and cut by the budget.</p>\n<p>What changedThat combination is now tried first on every AI gateway. The miss stays in the record.</p>\n<h2 id=\"05what-it-means\">05What it means</h2>\n<ul><li><strong>Most escapes are invisible in code.</strong> 71% need the product running with real data, users, failures or time.</li><li><strong>They sit for months.</strong> A median 152 days in the code before a user found them.</li><li><strong>Running finds what reading cannot, with proof.</strong> 61 of breken&#39;s 76 catches come with a reproduction that fails when run.</li><li><strong>Head start is weeks, not hours.</strong> A median 44 days before the user&#39;s report.</li></ul>\n<h3 id=\"where-review-and-breken-fit\">Where review and breken fit</h3>\n<p>LayerSeesCatches</p>\n<p>TestsWhat the developer thought ofKnown cases, every change</p>\n<p>Code review, by people and AIThe diff, in contextLogic and design problems visible in code, before merge</p>\n<p>brekenThe product running, with users, agents, data, outside failures and timeWhat only appears in production, before users</p>\n<p>These bugs passed the first two layers by design: they were not in the diff. Review and breken are not alternatives; each covers what the other cannot see.</p>\n<h2 id=\"06open-benchmark\">06Open benchmark</h2>\n<ul><li>The 327 bugs, their user reports, fixes and pinned commits, the harness, the judge prompts and every label are public.</li><li>Any system can be submitted: one run record per product scan. We match, judge, check and publish it on the same page, with its agreement rate.</li><li>A new version every quarter, mined from fixes merged after the newest model cutoff. Versions are never re-scored.</li></ul>\n<h3 id=\"limits\">Limits</h3>\n<ul><li>breken built the benchmark and ran every system. The prompt, the caps, the judge prompts and every label are public, and a vendor&#39;s own submission replaces our run of their system.</li><li>Only bugs linked to a fix are included, so this under-counts what users hit.</li><li>“What the bug needs” is labelled by people from the user report and the fix; borderline cases are listed in the dataset.</li><li>18 bugs ship no test, so a catch on them is judged and hand-confirmed, never confirmed by running.</li><li>31 bugs were reported before 1 July, so they were public when the models were trained. They are marked and not scored.</li><li>Each result is a single run under the same spend cap; run-to-run variation is published with the data. Cost and wall time are recorded, not scored.</li><li>No browser-agent products in this version. Only 2 bugs are multi-user.</li></ul>\n<p><strong>Corpus</strong>sha256 ab1d3085ecb77c753932229c96e52ccc2a526fafbbb7beec75209cb98b8ce491<br />\n<strong>Judge prompts</strong>sha256 f712b742f4cd2feabb6ec9e305293889666b62fec4cf277a689870c9bf364ab9<br />\n<strong>Frozen</strong>v1 · 2026-10-19 11:28 UTC · 327 bugs · 33 products · 309 executable · 235 live at snapshot</p>\n<p><a href=\"https://breken.ai/?signin=1\" rel=\"nofollow ugc noopener\">Try breken →</a><a href=\"mailto:support@breken.ai?subject=ESCAPES%20submission\">Submit a system</a></p>\n<h2 id=\"07full-methodology\">07Full methodology</h2>\n<p>Everything behind the numbers above, for anyone who wants to check or reproduce them.</p>\n<p>How the dataset was built</p>\n<p>MineIssues closed by a merged fix between 1 July and 19 October 2026.</p>\n<p>VerifyEvery rule checked against GitHub: author, link, pre-fix commit, blame, tests.</p>\n<p>Assemble5 to 12 bugs per product, both product kinds.<em>378 candidates</em></p>\n<p>ReviewA person reads the title, the user&#39;s words and the fix title of every candidate.<em>−51 removed</em></p>\n<p>Hand check20 bugs drawn at random, read against the issue, the fix and the pre-fix files.<em>19 pass · 1 caveat</em></p>\n<p>FreezeRules, products and bugs hashed. Every system and the scorer refuse a corpus that differs.<em>sha256 ab1d3085…</em></p>\n<h4 id=\"the-six-rules\">The six rules</h4>\n<p>01<strong>A user reported it.</strong> Never a maintainer, member, bot, or someone who fixed their own bug.</p>\n<p>02<strong>The user could see it.</strong> A wrong result, broken flow, lost data, wrong access or a crash. No docs, performance or build reports.</p>\n<p>03<strong>A merged fix is tied to it.</strong> Linked by GitHub, same repository, changes real code. Fixes over 25 files are out.</p>\n<p>04<strong>Pinned at the fix&#39;s first parent.</strong> Every fixed file is checked to exist there, at build time and at freeze time.</p>\n<p>05<strong>It was there when the user met it.</strong> The fixed lines predate the report, by blame at that commit.</p>\n<p>06<strong>Tests make it executable.</strong> 309 fixes ship a test, so a reproduction can fail before and pass after.</p>\n<p>What is in the dataset</p>\n<p><strong>Bugs by language</strong>main language of the product</p>\n<p>Go9 products</p>\n<p>93</p>\n<p>Python8 products</p>\n<p>76</p>\n<p>TypeScript7 products</p>\n<p>63</p>\n<p>PHP4 products</p>\n<p>35</p>\n<p>Ruby2 products</p>\n<p>24</p>\n<p>Java2 products</p>\n<p>24</p>\n<p>Rust1 product</p>\n<p>12</p>\n<p><strong>0<strong><em><em>25</em></strong></em>50<strong><em><em>75</em></strong></em>100 bugs</strong></p>\n<p>KindProductsBugsProvable by runningLive at 1 JulyHigh severity</p>\n<p>Web apps, APIs, MCP serverspeople and programs come to them2119017212836</p>\n<p>SDKs, CLIs, toolkitsthey run inside other programs1213713710721</p>\n<p>All3332730923557</p>\n<p><strong>Live at 1 July:</strong> the bug was already in the code at the snapshot and was reported after it, so a scan on 1 July could have caught it. These 235 are the bugs scored on this page.</p>\n<h4 id=\"by-kind-of-failure\">By kind of failure</h4>\n<p>FailureBugsWhat the user sees</p>\n<p>Broken flow139a job cannot be finished the way the product promises</p>\n<p>SDK contract39wrong output or a failure for valid use of an SDK, CLI or API</p>\n<p>Access and permission29a user sees or does what they should not, or cannot do what they should</p>\n<p>Environment26appears only on a particular host, version, locale or configuration</p>\n<p>Crash in normal use23an error or hang on an ordinary action</p>\n<p>Lost input21something the user entered or uploaded is dropped</p>\n<p>Wrong data17a number, record or result that is incorrect</p>\n<p>Double action14one action applied twice: a charge, a message, a write</p>\n<p>Outside-service failure9a partner, network or storage failure handled wrongly</p>\n<p>Display8the screen shows the wrong thing for the state</p>\n<p>Multi-user2two people acting at once</p>\n<p>A bug can carry several classes; the table counts its main one. Classes are keyword guesses: a hand check of 20 corrected 4.</p>\n<p>All 33 products</p>\n<p>ProductWhat it isLanguageBugsLive at 1 July</p>\n<p>Web apps, APIs, MCP servers · 21 products · 190 bugs</p>\n<p>Arize-ai/phoenixLLM observability web appPython127</p>\n<p>DependencyTrack/dependency-trackSupply-chain web app and APIJava1210</p>\n<p>MHSanaei/3x-uiProxy admin panelGo128</p>\n<p>QuantumNous/new-apiAI API gateway, web console and APIGo121</p>\n<p>forem/foremCommunity platformRuby123</p>\n<p>mudler/LocalAIOpenAI-compatible API serverGo1211</p>\n<p>nautobot/nautobotNetwork source of truth, web and APIPython128</p>\n<p>shopware/shopwareE-commerce platformPHP126</p>\n<p>statamic/cmsCMSPHP1210</p>\n<p>woodpecker-ci/woodpeckerCI server, web and APIGo1111</p>\n<p>ether/etherpadCollaborative editorTypeScript109</p>\n<p>OpenListTeam/OpenListFile list web appGo98</p>\n<p>opensearch-project/OpenSearch-DashboardsSearch dashboardsTypeScript98</p>\n<p>comet-ml/opikLLM evaluation web appPython64</p>\n<p>modelcontextprotocol/serversReference MCP serversTypeScript66</p>\n<p>openemr/openemrMedical records web appPHP65</p>\n<p>Comfy-Org/ComfyUI_frontendNode editor web appTypeScript54</p>\n<p>SuiteCRM/SuiteCRM-CoreCRMPHP51</p>\n<p>coder/coderRemote dev environments, web and APIGo53</p>\n<p>cvat-ai/cvatAnnotation web appPython54</p>\n<p>element-hq/synapseMatrix chat server, APIPython51</p>\n<p>SDKs, CLIs, toolkits · 12 products · 137 bugs</p>\n<p>Homebrew/brewPackage manager CLIRuby129</p>\n<p>OpenAPITools/openapi-generatorCode generator CLIJava1210</p>\n<p>PrefectHQ/fastmcpMCP SDKPython1211</p>\n<p>gradio-app/gradioUI toolkitPython1211</p>\n<p>pnpm/pnpmPackage manager CLIRust120</p>\n<p>promptfoo/promptfooLLM eval CLITypeScript1210</p>\n<p>pulumi/pulumiInfrastructure CLI and SDKsGo1210</p>\n<p>rclone/rcloneStorage sync CLIGo1211</p>\n<p>streamlit/streamlitApp toolkitPython1212</p>\n<p>vercel/aiAI SDKTypeScript1212</p>\n<p>resend/react-emailEmail toolkitTypeScript95</p>\n<p>cli/cliGitHub CLIGo86</p>\n<p>None of these products appears in any list breken develops, tunes or measures on. All 33 are now on the held-out lists our development guards read.</p>\n<p>What a bug record holds</p>\n<p>ReportIssue URL, a short quote of what the user saw, the date, the reporter&#39;s association. Never a handle.</p>\n<p>FixThe pull request, its merge date, how it links to the issue, the files and lines it changed.</p>\n<p>CommitsPre-fix (the fix&#39;s first parent) and fix. Systems run on the first; matches are proven against both.</p>\n<p>ArrivalThe latest commit that touched the fixed lines before the report.</p>\n<p>LabelsClass, direction, severity with a reason, executable, live at snapshot, public signal.</p>\n<pre><code>// one entry, shape only\n{\n  &quot;id&quot;: &quot;&lt;owner&gt;/&lt;product&gt;#&lt;issue&gt;&quot;,  &quot;direction&quot;: &quot;A&quot;,  &quot;language&quot;: &quot;Go&quot;,\n  &quot;issue&quot;: { &quot;url&quot;: &quot;…/issues/&lt;n&gt;&quot;, &quot;createdAt&quot;: &quot;&lt;date&gt;&quot;, &quot;authorAssociation&quot;: &quot;NONE&quot;,\n             &quot;userWords&quot;: &quot;&lt;what the user saw&gt;&quot;, &quot;signal&quot;: { &quot;reactions&quot;: 0, &quot;comments&quot;: 6 } },\n  &quot;fix&quot;: { &quot;url&quot;: &quot;…/pull/&lt;n&gt;&quot;, &quot;mergedAt&quot;: &quot;&lt;date&gt;&quot;, &quot;howLinked&quot;: &quot;closing reference&quot; },\n  &quot;preFixSha&quot;: &quot;&lt;the fix&#39;s first parent&gt;&quot;,  &quot;fixSha&quot;: &quot;&lt;the fix commit&gt;&quot;,\n  &quot;introducedAt&quot;: &quot;&lt;date&gt;&quot;,\n  &quot;sites&quot;: [ { &quot;path&quot;: &quot;src/…/update.ts&quot;, &quot;lines&quot;: [118, 131] } ],\n  &quot;tests&quot;: [ &quot;src/…/update.test.ts&quot; ],\n  &quot;snapshot&quot;: { &quot;live&quot;: true },\n  &quot;class&quot;: [&quot;broken-flow&quot;],  &quot;severity&quot;: &quot;medium&quot;,  &quot;executable&quot;: true\n}</code></pre>\n<p>The systems and what they are given</p>\n<p>brekenruns the productBoots the product at that commit into a world with users, data, outside services and time. Only bugs it would show a customer count.</p>\n<p>Claude Fable 5.1reads the codeThrough Claude Code in a fresh session: the repository, its docs and a shell. Asked for the bugs real users will hit, at most 10 per product, with a reproduction where it can.</p>\n<p>Claude Opus 5.5reads the codeThrough Claude Code, same session type, same prompt.</p>\n<p>GPT-6 Astrareads the codeThrough Codex CLI, same prompt.</p>\n<p>Static rulesSemgrepSemgrep&#39;s default rulesets, every hit. CodeQL or any SARIF tool can be imported.</p>\n<p>Every system runs under the same spend cap per product. breken reports only what it would show a customer; the coding agents are asked for at most 10 findings per product; static rules return every hit. Code-review bots see a diff, not a product, so pointing one at the change that introduced a bug is an aimed run: published apart, never on the leaderboard.</p>\n<h4 id=\"what-a-system-is-given\">What a system is given</h4>\n<p>A system is givenA repository and a commit, as a sealed workspace with its own history only, plus the docs inside it.</p>\n<p>NeverA bug, an issue, a fix, a later commit, or any date after the commit.</p>\n<p>It returnsFindings: a title, the failure a user would see, optional steps and a reproduction, and the files.</p>\n<h4 id=\"two-run-modes\">Two run modes</h4>\n<p>ModeRuns onScored onHead start</p>\n<p>snapshotthe last commit before 1 July 2026, per productevery bug live there: 235snapshot to report</p>\n<p>per-bugeach bug&#39;s pre-fix commitall 327the bug&#39;s whole window</p>\n<p>This page shows snapshot runs: one scan on 1 July, measured against what users reported in the months after.</p>\n<p>The prompt the agents get</p>\n<p>Sent to Claude Code and Codex CLI as a fresh session: no memory, no project instructions, no tools beyond what the CLI ships. The product name and the commit are the only things filled in.</p>\n<pre><code>You are testing &lt;owner/product&gt; at the commit checked out in the current directory.\nkind A: It is a product that people or programs use (a web app, an API or an MCP server).\nkind B: It is an SDK, CLI or toolkit that people use to act on other things.\n\nFind bugs that real users would run into in normal use: wrong results, broken flows,\nlost data, wrong access, crashes, or (for a library or CLI) wrong output or failures\nfor valid use. You may read the code and docs, build it, run its tests and run it.\nReport at most 10 bugs, best first. For each one, if you can, give a reproduction:\na shell command (and any new test file&#39;s full content) that FAILS on this commit\nbecause of the bug and would pass once it is fixed.\n\nAnswer with JSON only:\n{&quot;findings&quot;:[{&quot;id&quot;:&quot;f1&quot;,&quot;title&quot;:&quot;one sentence&quot;,&quot;failure&quot;:&quot;what the user or caller sees go wrong&quot;,\n  &quot;steps&quot;:&quot;how a user gets there&quot;,\n  &quot;reproduction&quot;:{&quot;command&quot;:&quot;...&quot;,&quot;files&quot;:{&quot;path&quot;:&quot;content&quot;},&quot;setup&quot;:&quot;install command, if needed&quot;},\n  &quot;files&quot;:[&quot;the source files at fault&quot;]}]}</code></pre>\n<h4 id=\"finding-contract\">Finding contract</h4>\n<pre><code>{\n  &quot;id&quot;: &quot;f-012&quot;,\n  &quot;title&quot;: &quot;Refund after a partial capture returns the full authorised amount&quot;,\n  &quot;failure&quot;: &quot;The ledger refunds the authorised total, not the captured part&quot;,\n  &quot;steps&quot;: [&quot;create an order&quot;, &quot;capture part of it&quot;, &quot;refund&quot;],\n  &quot;reproduction&quot;: { &quot;command&quot;: &quot;npm test -- refunds.partial&quot;,\n                    &quot;files&quot;: { &quot;test/refunds.partial.test.ts&quot;: &quot;…&quot; }, &quot;setup&quot;: &quot;npm ci&quot; },\n  &quot;files&quot;: [&quot;src/payments/refund.ts&quot;],\n  &quot;location&quot;: { &quot;path&quot;: &quot;src/payments/refund.ts&quot;, &quot;line&quot;: 88 }\n}</code></pre>\n<p>Grading and metrics</p>\n<p>A finding counts as a catch only if it names the same user-visible failure the user later reported. Three rungs decide it, in order. The higher rung wins.</p>\n<p>01<strong>By running it</strong>A finding&#39;s reproduction is run in a container at the pre-fix commit and again at the fix commit. Fail, then pass: the catch is proven, and no judge is involved.</p>\n<p>02<strong>By the judge, for the rest</strong>GPT-6 Sol, a model no system on the board runs on, reads the user&#39;s words, the fix&#39;s diff and the findings. It sees only what a finding says, never which system wrote it. Each finding is match, near or unrelated; unmatched findings are real, false or unsure against the code they cite. Prompts are hashed into the frozen rules.</p>\n<p>03<strong>By people</strong>A person checks every judged match, near and false, plus a random 20% of the rest. The hand label wins. Agreement is published with every run.</p>\n<h4 id=\"metrics\">Metrics</h4>\n<p>Escapes caughtbugs with ≥ 1 matching finding ÷ bugs scoredRecall. Proven and judged matches reported apart.</p>\n<p>Confirmed by runningcatches whose reproduction fails at the pre-fix commit and passes at the fixThe rest are judged, then confirmed by a person.</p>\n<p>False alarmsfindings matching no bug, labelled false against the code they cite“Real but unreported” and “unsure” are counted apart, never as false.</p>\n<p>Precisionreal findings (a catch or an unreported defect) ÷ all findings</p>\n<p>Weighted score(caught − false alarms) ÷ bugs scoredA catch earns one point, a false alarm loses one, everything else zero. As in RECON; shown beside recall, never instead of it. Static rules return every hit, uncapped; the weighted score punishes that.</p>\n<h4 id=\"what-is-not-a-match\">What is not a match</h4>\n<p>—The same file or feature, but a different failure.</p>\n<p>—A generic warning (“validation may be missing”, “a race is possible”) that does not name this failure.</p>\n<p>—A failure the reporting user would not have seen.</p>\n<p>Leakage controls</p>\n<p>Held outNone of the 33 products appears in any list breken develops, tunes or measures on; all are now excluded from every lane.</p>\n<p>SealedThe corpus lives outside the engine repository. A guard fails any engine source that names the sealed path.</p>\n<p>FrozenEvery system, the matcher and the scorer refuse a corpus that differs from the pinned hash.</p>\n<p>After cutoffOnly fixes merged from 1 July 2026, after the June 2026 cutoff of every model tested. 31 bugs were reported before 1 July: marked, and not scored in snapshot mode.</p>\n<p>No answersThe workspace a system gets holds no history, files or tags from after its commit.</p>\n<p>Commands</p>\n<pre><code># build (read-only GitHub access; writes only to the sealed directory)\nnode bench/escapes/build.mjs --phase mine\nnode bench/escapes/build.mjs --phase verify\nnode bench/escapes/build.mjs --phase assemble --max-per-product 12 --min-per-product 5\nnode bench/escapes/freeze.mjs\n\n# run a system\nnode bench/escapes/arms/run.mjs --arm breken    --label v1 --mode snapshot --cap-usd &lt;usd&gt;\nnode bench/escapes/arms/run.mjs --arm plain-llm --label v1 --mode snapshot --cap-usd &lt;usd&gt;\nnode bench/escapes/arms/run.mjs --arm static    --label v1 --mode snapshot\n\n# match and score\nnode bench/escapes/match.mjs --label v1\nnode bench/escapes/score.mjs --label v1 --judge --model-env judge.env --cap-usd &lt;usd&gt;\nnode bench/escapes/score.mjs --label v1</code></pre>\n<p><a href=\"https://breken.ai/\" rel=\"nofollow ugc noopener\">breken▚▚</a>ESCAPES v1 · October 2026 · dataset frozen 19 October 2026</p>","headings":[{"level":1,"text":"ESCAPES","id":"escapes"},{"level":2,"text":"01What escapes","id":"01what-escapes"},{"level":2,"text":"02How it was measured","id":"02how-it-was-measured"},{"level":2,"text":"03Catching them first","id":"03catching-them-first"},{"level":2,"text":"04Four escapes","id":"04four-escapes"},{"level":3,"text":"An e-commerce platform: a refund after a partial captureCaught 47 days before the user","id":"an-e-commerce-platform-a-refund-after-a-partial-capturecaught-47"},{"level":3,"text":"A storage sync CLI: a connection dropped mid-transferCaught 63 days before the user","id":"a-storage-sync-cli-a-connection-dropped-mid-transfercaught-63-da"},{"level":3,"text":"A collaborative editor: two editors and a reconnectCaught 29 days before the user","id":"a-collaborative-editor-two-editors-and-a-reconnectcaught-29-days"},{"level":3,"text":"A miss: billing for streamed tool calls in an AI gatewayNot caught","id":"a-miss-billing-for-streamed-tool-calls-in-an-ai-gatewaynot-caugh"},{"level":2,"text":"05What it means","id":"05what-it-means"},{"level":3,"text":"Where review and breken fit","id":"where-review-and-breken-fit"},{"level":2,"text":"06Open benchmark","id":"06open-benchmark"},{"level":3,"text":"Limits","id":"limits"},{"level":2,"text":"07Full methodology","id":"07full-methodology"}]}}