---
title: "SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations"
slug: swe-cowork-benchmarking-coding-agents-as-digital-employees-in-organizations
url: https://listedarticles.com/articles/swe-cowork-benchmarking-coding-agents-as-digital-employees-in-organizations
canonical_url: https://arxiv.org/abs/2611.20218
content_type: research
language: en
published_at: 2026-11-15T00:00:00.000Z
updated_at: 2026-11-17T22:51:36.171Z
author: "Xinping Lei et al."
authored_by: human
publisher: "TokenWave AI"
publisher_url: https://tokenwave-ai.github.io/swe-cowork-page/
topics: ["ai-agents", "benchmarks", "research", "software-engineering"]
license: all-rights-reserved
word_count: 240
reading_minutes: 1
citation: "Xinping Lei et al., TokenWave AI. \"SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations.\" 15 Nov 2026. https://arxiv.org/abs/2611.20218 (all-rights-reserved)"
---

# SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

> TokenWave AI's SWE-CoWork benchmark drops coding agents into simulated engineering orgs with 26 colleagues across 101 real repositories; the two leading models pass about 63% of target tests but complete fewer than 8% of assignments end to end.

## Abstract

A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.

---

Source: [arXiv:2611.20218](https://arxiv.org/abs/2611.20218) (licensed [CC BY-NC-SA 4.0](http://creativecommons.org/licenses/by-nc-sa/4.0/)). Project page: [tokenwave-ai.github.io/swe-cowork-page](https://tokenwave-ai.github.io/swe-cowork-page/). Code: [TokenWave-AI/swe-cowork-bench](https://github.com/TokenWave-AI/swe-cowork-bench).
