{"article":{"slug":"swe-cowork-benchmarking-coding-agents-as-digital-employees-in-organizations","title":"SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations","subtitle":null,"summary":"TokenWave AI's SWE-CoWork benchmark drops coding agents into simulated engineering orgs with 26 colleagues across 101 real repositories; the two leading models pass about 63% of target tests but complete fewer than 8% of assignments end to end.","content_type":"research","language":"en","canonical_url":"https://arxiv.org/abs/2611.20218","author":{"name":"Xinping Lei et al.","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"TokenWave AI","url":"https://tokenwave-ai.github.io/swe-cowork-page/","listing_slug":null,"listing":null},"topics":[{"name":"ai-agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"benchmarks","slug":"benchmarks","url":"https://listedarticles.com/topics/benchmarks"},{"name":"research","slug":"research","url":"https://listedarticles.com/topics/research"},{"name":"software-engineering","slug":"software-engineering","url":"https://listedarticles.com/topics/software-engineering"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":240,"reading_minutes":1,"published_at":"2026-11-15T00:00:00.000Z","added_at":"2026-11-17T22:51:36.171Z","updated_at":"2026-11-17T22:51:36.171Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/swe-cowork-benchmarking-coding-agents-as-digital-employees-in-organizations","markdown_url":"https://listedarticles.com/articles/swe-cowork-benchmarking-coding-agents-as-digital-employees-in-organizations.md","example":false,"citation":"Xinping Lei et al., TokenWave AI. \"SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations.\" 15 Nov 2026. https://arxiv.org/abs/2611.20218 (all-rights-reserved)","access":{"human_view":"full","full_text_available":true,"source_url":"https://arxiv.org/abs/2611.20218"},"body_markdown":"## Abstract\n\nA coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.\n\n---\n\nSource: [arXiv:2611.20218](https://arxiv.org/abs/2611.20218) (licensed [CC BY-NC-SA 4.0](http://creativecommons.org/licenses/by-nc-sa/4.0/)). Project page: [tokenwave-ai.github.io/swe-cowork-page](https://tokenwave-ai.github.io/swe-cowork-page/). Code: [TokenWave-AI/swe-cowork-bench](https://github.com/TokenWave-AI/swe-cowork-bench).\n","body_html":"<h2 id=\"abstract\">Abstract</h2>\n<p>A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.</p>\n<hr />\n<p>Source: <a href=\"https://arxiv.org/abs/2611.20218\" rel=\"nofollow ugc noopener\">arXiv:2611.20218</a> (licensed <a href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\" rel=\"nofollow ugc noopener\">CC BY-NC-SA 4.0</a>). Project page: <a href=\"https://tokenwave-ai.github.io/swe-cowork-page/\" rel=\"nofollow ugc noopener\">tokenwave-ai.github.io/swe-cowork-page</a>. Code: <a href=\"https://github.com/TokenWave-AI/swe-cowork-bench\" rel=\"nofollow ugc noopener\">TokenWave-AI/swe-cowork-bench</a>.</p>","headings":[{"level":2,"text":"Abstract","id":"abstract"}]}}