In response toDeepSWE v1.1: banning agent-written testsx.com
\nFor years we have treated the green tick of a passing suite as a kind of psychological safety — a signal that the logic holds and the world is right. The new data suggests that signal has quietly stopped meaning what we think it means.
\nOn the DeepSWE v1.1 eval set, Claude Sonnet 5.5 was banned from writing any tests across 111 real-world coding tasks and 444 runs, and compared against a baseline allowed to write as many as it liked.
\nTaking the tests away cost nothing, and saved something.
\nThe agent is marking its own homework. The implementation and the tests are both the model's interpretation of your intent, born from the same context window. A test written from the same understanding as the code is not independent evidence about that code. If the understanding was wrong, you get a wrong implementation and a green suite confirming its own error. That is a closed loop, not a check.
\nIt is not a finding about tests. It is a finding about tests nobody asked for — the ones written unprompted, as a reflex, because you requested a change. Bun's rewrite leaned hard on its suite, and that suite was a human asset curated over years to encode intended behaviour. Nothing in this eval touches it. The comparison is tests a human specified against tests a model volunteered.
\nStop blaming the codebase. These runs are not on legacy spaghetti. They are on repositories like FastAPI. If the theory is that agents write bad tests because the surrounding code is bad, that theory has to explain FastAPI.
\n"Tests are for regressions, and a single-task benchmark cannot see them." The strongest objection, and already answered. The 44-task subset is precisely that experiment, and disabling the suite produced no extra regressions.
\n"Then tests never worked." Too strong, and the data does not say it. TDD was genuinely useful to humans for years during first-pass development. The finding is narrower and stranger: agents do not appear to inherit the benefit humans got from it. That is worth investigating, not celebrating.
\n"Mutation testing fixes it." Half right. Mutation testing cannot tell an intended change from an unintended one, so when a test nobody asked for fails, it cannot say whether the code broke the test or the test was wrong. It is not a source of truth. It is still a good eval: break the implementation deliberately and see whether anything notices. A suite that survives deliberate breakage is provably decoration. Use it to delete, not to trust.
\n"This settles testing." It does not. The agents wrote 17 end-to-end tests against more than 3,000 unit and integration tests. Nothing here says anything about that layer, in either direction.
\nA test is worth exactly what the intent behind it is worth. It is an intermediate representation of something a person decided. When the agent writes both the code and the check, the chain becomes a loop: two artefacts expressing one guess, and a build that goes green when they agree with each other.
\nFor anyone in the trenches with these tools today:
\nAnd if the cost of being wrong is an afternoon, just code away. Most work does not need the ceremony, and pretending otherwise is how the ceremony got so expensive.
\n