{"article":{"slug":"brood-war-bench","title":"Brood War Bench","subtitle":null,"summary":"Ben Swerdlow benchmarks Codex, Claude, and Grok agents across 171 StarCraft: Brood War matches—leaderboards, APM, cost per game, and analysis showing none played beyond beginner while Codex Astra led.","content_type":"article","language":"en","canonical_url":"https://bw.swerdlow.dev/report","author":{"name":"Ben Swerdlow","url":"https://twitter.com/benswerd","person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Ben Swerdlow","url":null,"listing_slug":null,"listing":null},"topics":[{"name":"AI","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"Benchmarks","slug":"benchmarks","url":"https://listedarticles.com/topics/benchmarks"},{"name":"AI Agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"Games","slug":"games","url":"https://listedarticles.com/topics/games"}],"about_listings":[{"slug":"freestyle","name":"Freestyle","listing_type":"company","url":"https://listedstartups.com/companies/freestyle"}],"cover_image_url":null,"license":"all-rights-reserved","word_count":2151,"reading_minutes":9,"published_at":"2026-09-19T00:00:00.000Z","added_at":"2026-09-19T21:09:46.410Z","updated_at":"2026-09-19T21:09:46.410Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":false},"profile_url":"https://listedarticles.com/articles/brood-war-bench","markdown_url":"https://listedarticles.com/articles/brood-war-bench.md","example":false,"citation":"Ben Swerdlow, Ben Swerdlow. \"Brood War Bench.\" 19 Sept 2026. https://bw.swerdlow.dev/report (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://bw.swerdlow.dev/report"},"body_markdown":"# Brood War Bench\n\nBy [Ben Swerdlow]()\n\nWhich model wins at Brood War?\n\n## Key takeaways\n\n * None of the models played beyond a beginner level.\n * Codex Astra is the clear leader beating all other models consistently.\n * Grok models are not smart enough to play Brood War yet.\n * Older models tended to play the RTS as a turn-based game, leading them to get destroyed while they were thinking. Newer models sometimes fell into the same trap, which may explain why some lower-effort settings performed better, but overall were much more cognizant of the cost of thinking.\n\nPowered By [Freestyle]()\n\n## Leaderboard\n\nRank| System| Wins| Losses| APM| Cost / game| Win rate \n---|---|---|---|---|---|--- \n🥇| **Codex Astra / xhigh**| 18| 0| 12.6| $10.54| **100.0%** \n🥈| **Codex Astra / medium**| 16| 2| 17.2| $15.11| **88.9%** \n🥉| **Claude Fable**| 15| 3| 12.6| $12.24| **83.3%** \n4| **Codex Astra / low**| 14| 4| 25.7| $21.07| **77.8%** \n5| **Codex 5.6 Sol / medium**| 13| 5| 10.1| $5.12| **72.2%** \n6| **Codex 5.6 Sol / low**| 12| 6| 18.1| $9.23| **66.7%** \n7| **Claude Opus 5**| 12| 6| 10.5| $20.78| **66.7%** \n8| **Codex 5.6 Sol / xhigh**| 11| 7| 8.0| $3.23| **61.1%** \n9| **Codex 5.6 Luna / low**| 9| 9| 23.8| $0.42| **50.0%** \n10| **Codex 5.6 Terra / xhigh**| 9| 9| 15.8| $2.10| **50.0%** \n11| **Codex 5.6 Terra / medium**| 8| 10| 10.5| $3.15| **44.4%** \n12| **Codex 5.6 Terra / low**| 8| 10| 48.3| $4.65| **44.4%** \n13| **Codex 5.6 Luna / xhigh**| 7| 11| 5.2| $0.16| **38.9%** \n14| **Claude Sonnet**| 7| 11| 6.2| $8.98| **38.9%** \n15| **Codex 5.6 Luna / medium**| 6| 12| 14.8| $0.30| **33.3%** \n16| **Grok 4.6 / xhigh**| 2| 15| 2.8| $0.66| **11.1%** \n17| **Grok 4.6 / medium**| 1| 16| 3.2| $0.79| **5.6%** \n18| **Claude Haiku**| 0| 16| 0.3| $0.34| **0.0%** \n19| **Grok 4.6 / low**| 0| 16| 4.2| $1.26| **0.0%** \n \nShow all\n\nBrood War Bench started after I built a version of Brood War that you could only play through agents as an experiment to play with friends. I played it with a couple friends who did surprisingly well for people who have only played a couple Starcraft games in their lives. When I asked them why, they said they hadn't done much, they asked their agent to attack and it had built a small army and done the full attack for them. This lead me to wonder how far they can go on their own; this is my answer.\n\n## What I observed\n\n01\n\n### Codex found cheese before it found macro\n\nCodex's strongest recurring idea was disruption. In Protoss games it often sent a Probe across the map to attack workers or buildings. This worked shockingly well as the opposing agents often spent dozens of seconds thinking about what to do about a probe instead of doing anything else.\n\nThe same systems were much weaker at sustained production. They delayed tech, trickled one or two basic units into defended bases, and threw workers into last stands.\n\nI also noticed Codex often created separate subagents to manage the economy, army production, and army control. They didn't communicate much with one another, so the army agent often sent each new unit straight into an attack, unaware of the larger army the other agents were planning to build.\n\nThis is a common beginner mistake: sending units in one at a time instead of waiting for a critical mass and a planned attack timing. In games where I helped direct Codex, it was much better at planning those moments and getting its subagents to work together.\n\nThe persistence was real. In G009, after losing its army and main base, Codex 5.6 Terra / medium lifted its last Command Center and moved it toward the opposite corner. It survived for another six minutes.\n\nSix Probes cross the mapA Probe first, then Zealots in dripsThe last Command Center runs\n\n02\n\n### Grok spent the game between actions\n\nGrok 4.6 frequently produced long stretches of reasoning and very few command batches. In G043, the xhigh run logged 11,138 reasoning tokens but issued only six command batches across 43 minutes and never fielded a combat unit.\n\nThe actions it did take rarely developed into a working control loop. In G003, Grok / xhigh made three Marines and never reached the enemy base. In G002, Grok / medium made two Zealots and also never crossed the map. These looked less like bad strategies than failures to keep observing and acting.\n\nForty-three minutes, no army\n\n03\n\n### Fable earnestly tried to play the game\n\nI found myself rooting for Claude Fable in more than a few games. Fable usually tried to build an economy and climb the tech tree instead of stopping at the first unit available. It seemed more interested in actually playing the game than any of the other models.\n\nIn G007 it reached a Lair, Spire, and Mutalisks and won. In G027 it added a Robotics Facility, Citadel of Adun, Observatory, and Templar Archives before winning. Ambition did not guarantee execution: in G036 Fable reached a Factory and Academy but Opus 5 overran it.\n\nFable gets MutalisksFable keeps climbingThe build does not become an army\n\n### No agent here played beyond beginner level\n\nEven Astra and Fable were unable to build complex army's, defend simple attacks or play concrete strategies. A beginner playing photon rush would win every single one of these games.\n\nThat said, watching the agents play made me more excited than I have been in a while. This benchmark is nowhere near exhausted. There is much more for the agents to learn, and much more for the benchmark to ask them to do. I look forward to watching them get there.\n\n## How the games developed\n\nCompareModelsModel + effortHarnessesRaceAll racesTerranProtossZergGame timeFirst 10 minutesFirst 15 minutesFirst 20 minutesFull games\n\nChoose up to five\n\nCodex AstraClaude FableCodex 5.6 SolClaude Opus 5Codex 5.6 LunaCodex 5.6 TerraClaude SonnetGrok 4.6Claude Haiku\n\n**5:00** / 15:00\n\n### Technology investment\n\nCompleted research + upgrade levels\n\n00.511.520:005:0010:0015:00\n\n __Codex Astra**0.3** __Claude Fable**0.2** __Grok 4.6**0**\n\n### Workers\n\nCompleted workers alive\n\n081624320:005:0010:0015:00\n\n __Codex Astra**14.7** __Claude Fable**16.7** __Grok 4.6**7.4**\n\n### Army size\n\nCompleted army and support units\n\n0204060800:005:0010:0015:00\n\n __Codex Astra**8.3** __Claude Fable**7.6** __Grok 4.6**2**\n\n### Structures\n\nCompleted buildings, including add-ons\n\n071421280:005:0010:0015:00\n\n __Codex Astra**6.2** __Claude Fable**7.4** __Grok 4.6**4.5**\n\n### Minerals in the bank\n\nUnspent minerals, not income\n\n05001,0001,5002,0000:005:0010:0015:00\n\n __Codex Astra**240.6** __Claude Fable**248.6** __Grok 4.6**536.6**\n\n### Gas in the bank\n\nUnspent gas, not income\n\n05001,0001,5002,0000:005:0010:0015:00\n\n __Codex Astra**151** __Claude Fable**326.9** __Grok 4.6**244.5**\n\n### Supply used\n\nIncludes production in progress\n\n03060901200:005:0010:0015:00\n\n __Codex Astra**26.5** __Claude Fable**26.6** __Grok 4.6**11.3**\n\n### When games ended\n\nShare of games ending per 5-minute window\n\nCodex Astra: median 8:10. 0:00 to before 5:00: 3.9%; 5:00 to before 10:00: 54.9%; 10:00 to before 15:00: 31.4%; 15:00 to before 20:00: 5.9%; 25:00 to before 30:00: 2%; 30:00 to before 35:00: 2%. Claude Fable: median 10:37. 5:00 to before 10:00: 38.9%; 10:00 to before 15:00: 38.9%; 15:00 to before 20:00: 22.2%. Grok 4.6: median 9:15. 0:00 to before 5:00: 2%; 5:00 to before 10:00: 51%; 10:00 to before 15:00: 23.5%; 15:00 to before 20:00: 13.7%; 20:00 to before 25:00: 2%; 40:00 to before 45:00: 7.8%.Codex AstraMedian 8:1060%Codex Astra: 3.9% (2 games) ended from 0:00 to before 5:00Codex Astra: 54.9% (28 games) ended from 5:00 to before 10:00Codex Astra: 31.4% (16 games) ended from 10:00 to before 15:00Codex Astra: 5.9% (3 games) ended from 15:00 to before 20:00Codex Astra: 0% (0 games) ended from 20:00 to before 25:00Codex Astra: 2% (1 games) ended from 25:00 to before 30:00Codex Astra: 2% (1 games) ended from 30:00 to before 35:00Codex Astra: 0% (0 games) ended from 35:00 to before 40:00Codex Astra: 0% (0 games) ended from 40:00 to before 45:00Claude FableMedian 10:3760%Claude Fable: 0% (0 games) ended from 0:00 to before 5:00Claude Fable: 38.9% (7 games) ended from 5:00 to before 10:00Claude Fable: 38.9% (7 games) ended from 10:00 to before 15:00Claude Fable: 22.2% (4 games) ended from 15:00 to before 20:00Claude Fable: 0% (0 games) ended from 20:00 to before 25:00Claude Fable: 0% (0 games) ended from 25:00 to before 30:00Claude Fable: 0% (0 games) ended from 30:00 to before 35:00Claude Fable: 0% (0 games) ended from 35:00 to before 40:00Claude Fable: 0% (0 games) ended from 40:00 to before 45:00Grok 4.6Median 9:1560%Grok 4.6: 2% (1 games) ended from 0:00 to before 5:00Grok 4.6: 51% (26 games) ended from 5:00 to before 10:00Grok 4.6: 23.5% (12 games) ended from 10:00 to before 15:00Grok 4.6: 13.7% (7 games) ended from 15:00 to before 20:00Grok 4.6: 2% (1 games) ended from 20:00 to before 25:00Grok 4.6: 0% (0 games) ended from 25:00 to before 30:00Grok 4.6: 0% (0 games) ended from 30:00 to before 35:00Grok 4.6: 0% (0 games) ended from 35:00 to before 40:00Grok 4.6: 7.8% (4 games) ended from 40:00 to before 45:000:0015:0030:0045:00\n\nTime-series charts show means of recorded player-runs at each game time. Finished games drop out; missing samples are not filled. Models pool their effort settings. Units and buildings count only once completed; army excludes workers, Overlords, eggs, larvae, and ammunition.\n\n## Win rate vs. cost\n\nLog scaleLinear\n\nAverage cost per game, using the same prices as the leaderboard. Codex and Sonnet costs are token-based estimates.\n\nCodexClaudeGrok\n\nWin rate0%25%50%75%100%$0.1$0.5$1$5$10$20Cost per game (USD, log scale)\n\n## How the benchmark ran\n\nWe built a round-robin matrix of model and effort configurations and had every configuration play every other. The harness ran those matchups in parallel across Freestyle VMs, saving game-engine data and both agents' harness logs for each match.\n\n## Head-to-head matrix\n\nRead across a row. W is a win, L is a loss, and T is a match that reached the benchmark time limit.\n\nOpen the full 19 × 19 matrix\n\nW win L loss T time limit G000pending\n\nSystem| 1| 2| 3| 4| 5| 6| 7| 8| 9| 10| 11| 12| 13| 14| 15| 16| 17| 18| 19 \n---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|--- \n1**Codex Astra / xhigh**| -| W| W| W| W| W| W| W| W| W| W| W| W| W| W| W| W| W| W \n2**Codex Astra / medium**| L| -| W| W| W| W| W| W| W| W| W| W| L| W| W| W| W| W| W \n3**Codex Astra / low**| L| L| -| W| W| W| W| W| W| W| W| W| L| L| W| W| W| W| W \n4**Codex 5.6 Sol / xhigh**| L| L| L| -| L| W| W| W| L| W| W| W| L| L| W| W| W| W| W \n5**Codex 5.6 Sol / medium**| L| L| L| W| -| W| W| W| L| W| W| W| L| W| W| W| W| W| W \n6**Codex 5.6 Sol / low**| L| L| L| L| L| -| W| W| W| W| W| W| W| W| L| W| W| W| W \n7**Codex 5.6 Luna / xhigh**| L| L| L| L| L| L| -| W| L| W| L| W| L| L| L| W| W| W| W \n8**Codex 5.6 Luna / medium**| L| L| L| L| L| L| L| -| L| L| L| L| L| W| W| W| W| W| W \n9**Codex 5.6 Luna / low**| L| L| L| W| W| L| W| W| -| W| L| L| L| L| L| W| W| W| W \n10**Codex 5.6 Terra / xhigh**| L| L| L| L| L| L| L| W| L| -| W| W| L| W| W| W| W| W| W \n11**Codex 5.6 Terra / medium**| L| L| L| L| L| L| W| W| W| L| -| L| L| L| W| W| W| W| W \n12**Codex 5.6 Terra / low**| L| L| L| L| L| L| L| W| W| L| W| -| L| L| W| W| W| W| W \n13**Claude Fable**| L| W| W| W| W| L| W| W| W| W| W| W| -| L| W| W| W| W| W \n14**Claude Opus 5**| L| L| W| W| L| L| W| L| W| L| W| W| W| -| W| W| W| W| W \n15**Claude Sonnet**| L| L| L| L| L| W| W| L| W| L| L| L| L| L| -| W| W| W| W \n16**Claude Haiku**| L| L| L| L| L| L| L| L| L| L| L| L| L| L| L| -| L| T| T \n17**Grok 4.6 / xhigh**| L| L| L| L| L| L| L| L| L| L| L| L| L| L| L| W| -| W| T \n18**Grok 4.6 / medium**| L| L| L| L| L| L| L| L| L| L| L| L| L| L| L| T| L| -| W \n19**Grok 4.6 / low**| L| L| L| L| L| L| L| L| L| L| L| L| L| L| L| T| T| L| - \n \n## Play your own match\n\nBring your agent and play Brood War with friends.\n\n[Play Brood War]()","body_html":"

Brood War Bench

\n

By Ben Swerdlow

\n

Which model wins at Brood War?

\n

Key takeaways

\n\n

Powered By Freestyle

\n

Leaderboard

\n
RankSystemWinsLossesAPMCost / gameWin rate
🥇Codex Astra / xhigh18012.6$10.54100.0%
🥈Codex Astra / medium16217.2$15.1188.9%
🥉Claude Fable15312.6$12.2483.3%
4Codex Astra / low14425.7$21.0777.8%
5Codex 5.6 Sol / medium13510.1$5.1272.2%
6Codex 5.6 Sol / low12618.1$9.2366.7%
7Claude Opus 512610.5$20.7866.7%
8Codex 5.6 Sol / xhigh1178.0$3.2361.1%
9Codex 5.6 Luna / low9923.8$0.4250.0%
10Codex 5.6 Terra / xhigh9915.8$2.1050.0%
11Codex 5.6 Terra / medium81010.5$3.1544.4%
12Codex 5.6 Terra / low81048.3$4.6544.4%
13Codex 5.6 Luna / xhigh7115.2$0.1638.9%
14Claude Sonnet7116.2$8.9838.9%
15Codex 5.6 Luna / medium61214.8$0.3033.3%
16Grok 4.6 / xhigh2152.8$0.6611.1%
17Grok 4.6 / medium1163.2$0.795.6%
18Claude Haiku0160.3$0.340.0%
19Grok 4.6 / low0164.2$1.260.0%
\n

Show all

\n

Brood War Bench started after I built a version of Brood War that you could only play through agents as an experiment to play with friends. I played it with a couple friends who did surprisingly well for people who have only played a couple Starcraft games in their lives. When I asked them why, they said they hadn't done much, they asked their agent to attack and it had built a small army and done the full attack for them. This lead me to wonder how far they can go on their own; this is my answer.

\n

What I observed

\n

01

\n

Codex found cheese before it found macro

\n

Codex's strongest recurring idea was disruption. In Protoss games it often sent a Probe across the map to attack workers or buildings. This worked shockingly well as the opposing agents often spent dozens of seconds thinking about what to do about a probe instead of doing anything else.

\n

The same systems were much weaker at sustained production. They delayed tech, trickled one or two basic units into defended bases, and threw workers into last stands.

\n

I also noticed Codex often created separate subagents to manage the economy, army production, and army control. They didn't communicate much with one another, so the army agent often sent each new unit straight into an attack, unaware of the larger army the other agents were planning to build.

\n

This is a common beginner mistake: sending units in one at a time instead of waiting for a critical mass and a planned attack timing. In games where I helped direct Codex, it was much better at planning those moments and getting its subagents to work together.

\n

The persistence was real. In G009, after losing its army and main base, Codex 5.6 Terra / medium lifted its last Command Center and moved it toward the opposite corner. It survived for another six minutes.

\n

Six Probes cross the mapA Probe first, then Zealots in dripsThe last Command Center runs

\n

02

\n

Grok spent the game between actions

\n

Grok 4.6 frequently produced long stretches of reasoning and very few command batches. In G043, the xhigh run logged 11,138 reasoning tokens but issued only six command batches across 43 minutes and never fielded a combat unit.

\n

The actions it did take rarely developed into a working control loop. In G003, Grok / xhigh made three Marines and never reached the enemy base. In G002, Grok / medium made two Zealots and also never crossed the map. These looked less like bad strategies than failures to keep observing and acting.

\n

Forty-three minutes, no army

\n

03

\n

Fable earnestly tried to play the game

\n

I found myself rooting for Claude Fable in more than a few games. Fable usually tried to build an economy and climb the tech tree instead of stopping at the first unit available. It seemed more interested in actually playing the game than any of the other models.

\n

In G007 it reached a Lair, Spire, and Mutalisks and won. In G027 it added a Robotics Facility, Citadel of Adun, Observatory, and Templar Archives before winning. Ambition did not guarantee execution: in G036 Fable reached a Factory and Academy but Opus 5 overran it.

\n

Fable gets MutalisksFable keeps climbingThe build does not become an army

\n

No agent here played beyond beginner level

\n

Even Astra and Fable were unable to build complex army's, defend simple attacks or play concrete strategies. A beginner playing photon rush would win every single one of these games.

\n

That said, watching the agents play made me more excited than I have been in a while. This benchmark is nowhere near exhausted. There is much more for the agents to learn, and much more for the benchmark to ask them to do. I look forward to watching them get there.

\n

How the games developed

\n

CompareModelsModel + effortHarnessesRaceAll racesTerranProtossZergGame timeFirst 10 minutesFirst 15 minutesFirst 20 minutesFull games

\n

Choose up to five

\n

Codex AstraClaude FableCodex 5.6 SolClaude Opus 5Codex 5.6 LunaCodex 5.6 TerraClaude SonnetGrok 4.6Claude Haiku

\n

5:00 / 15:00

\n

Technology investment

\n

Completed research + upgrade levels

\n

00.511.520:005:0010:0015:00

\n

__Codex Astra0.3 __Claude Fable0.2 __Grok 4.6*0*

\n

Workers

\n

Completed workers alive

\n

081624320:005:0010:0015:00

\n

__Codex Astra14.7 __Claude Fable16.7 __Grok 4.67.4

\n

Army size

\n

Completed army and support units

\n

0204060800:005:0010:0015:00

\n

__Codex Astra8.3 __Claude Fable7.6 __Grok 4.6*2*

\n

Structures

\n

Completed buildings, including add-ons

\n

071421280:005:0010:0015:00

\n

__Codex Astra6.2 __Claude Fable7.4 __Grok 4.64.5

\n

Minerals in the bank

\n

Unspent minerals, not income

\n

05001,0001,5002,0000:005:0010:0015:00

\n

__Codex Astra240.6 __Claude Fable248.6 __Grok 4.6536.6

\n

Gas in the bank

\n

Unspent gas, not income

\n

05001,0001,5002,0000:005:0010:0015:00

\n

__Codex Astra151 __Claude Fable326.9 __Grok 4.6244.5

\n

Supply used

\n

Includes production in progress

\n

03060901200:005:0010:0015:00

\n

__Codex Astra26.5 __Claude Fable26.6 __Grok 4.611.3

\n

When games ended

\n

Share of games ending per 5-minute window

\n

Codex Astra: median 8:10. 0:00 to before 5:00: 3.9%; 5:00 to before 10:00: 54.9%; 10:00 to before 15:00: 31.4%; 15:00 to before 20:00: 5.9%; 25:00 to before 30:00: 2%; 30:00 to before 35:00: 2%. Claude Fable: median 10:37. 5:00 to before 10:00: 38.9%; 10:00 to before 15:00: 38.9%; 15:00 to before 20:00: 22.2%. Grok 4.6: median 9:15. 0:00 to before 5:00: 2%; 5:00 to before 10:00: 51%; 10:00 to before 15:00: 23.5%; 15:00 to before 20:00: 13.7%; 20:00 to before 25:00: 2%; 40:00 to before 45:00: 7.8%.Codex AstraMedian 8:1060%Codex Astra: 3.9% (2 games) ended from 0:00 to before 5:00Codex Astra: 54.9% (28 games) ended from 5:00 to before 10:00Codex Astra: 31.4% (16 games) ended from 10:00 to before 15:00Codex Astra: 5.9% (3 games) ended from 15:00 to before 20:00Codex Astra: 0% (0 games) ended from 20:00 to before 25:00Codex Astra: 2% (1 games) ended from 25:00 to before 30:00Codex Astra: 2% (1 games) ended from 30:00 to before 35:00Codex Astra: 0% (0 games) ended from 35:00 to before 40:00Codex Astra: 0% (0 games) ended from 40:00 to before 45:00Claude FableMedian 10:3760%Claude Fable: 0% (0 games) ended from 0:00 to before 5:00Claude Fable: 38.9% (7 games) ended from 5:00 to before 10:00Claude Fable: 38.9% (7 games) ended from 10:00 to before 15:00Claude Fable: 22.2% (4 games) ended from 15:00 to before 20:00Claude Fable: 0% (0 games) ended from 20:00 to before 25:00Claude Fable: 0% (0 games) ended from 25:00 to before 30:00Claude Fable: 0% (0 games) ended from 30:00 to before 35:00Claude Fable: 0% (0 games) ended from 35:00 to before 40:00Claude Fable: 0% (0 games) ended from 40:00 to before 45:00Grok 4.6Median 9:1560%Grok 4.6: 2% (1 games) ended from 0:00 to before 5:00Grok 4.6: 51% (26 games) ended from 5:00 to before 10:00Grok 4.6: 23.5% (12 games) ended from 10:00 to before 15:00Grok 4.6: 13.7% (7 games) ended from 15:00 to before 20:00Grok 4.6: 2% (1 games) ended from 20:00 to before 25:00Grok 4.6: 0% (0 games) ended from 25:00 to before 30:00Grok 4.6: 0% (0 games) ended from 30:00 to before 35:00Grok 4.6: 0% (0 games) ended from 35:00 to before 40:00Grok 4.6: 7.8% (4 games) ended from 40:00 to before 45:000:0015:0030:0045:00

\n

Time-series charts show means of recorded player-runs at each game time. Finished games drop out; missing samples are not filled. Models pool their effort settings. Units and buildings count only once completed; army excludes workers, Overlords, eggs, larvae, and ammunition.

\n

Win rate vs. cost

\n

Log scaleLinear

\n

Average cost per game, using the same prices as the leaderboard. Codex and Sonnet costs are token-based estimates.

\n

CodexClaudeGrok

\n

Win rate0%25%50%75%100%$0.1$0.5$1$5$10$20Cost per game (USD, log scale)

\n

How the benchmark ran

\n

We built a round-robin matrix of model and effort configurations and had every configuration play every other. The harness ran those matchups in parallel across Freestyle VMs, saving game-engine data and both agents' harness logs for each match.

\n

Head-to-head matrix

\n

Read across a row. W is a win, L is a loss, and T is a match that reached the benchmark time limit.

\n

Open the full 19 × 19 matrix

\n

W win L loss T time limit G000pending

\n
System12345678910111213141516171819
1Codex Astra / xhigh-WWWWWWWWWWWWWWWWWW
2Codex Astra / mediumL-WWWWWWWWWWLWWWWWW
3Codex Astra / lowLL-WWWWWWWWWLLWWWWW
4Codex 5.6 Sol / xhighLLL-LWWWLWWWLLWWWWW
5Codex 5.6 Sol / mediumLLLW-WWWLWWWLWWWWWW
6Codex 5.6 Sol / lowLLLLL-WWWWWWWWLWWWW
7Codex 5.6 Luna / xhighLLLLLL-WLWLWLLLWWWW
8Codex 5.6 Luna / mediumLLLLLLL-LLLLLWWWWWW
9Codex 5.6 Luna / lowLLLWWLWW-WLLLLLWWWW
10Codex 5.6 Terra / xhighLLLLLLLWL-WWLWWWWWW
11Codex 5.6 Terra / mediumLLLLLLWWWL-LLLWWWWW
12Codex 5.6 Terra / lowLLLLLLLWWLW-LLWWWWW
13Claude FableLWWWWLWWWWWW-LWWWWW
14Claude Opus 5LLWWLLWLWLWWW-WWWWW
15Claude SonnetLLLLLWWLWLLLLL-WWWW
16Claude HaikuLLLLLLLLLLLLLLL-LTT
17Grok 4.6 / xhighLLLLLLLLLLLLLLLW-WT
18Grok 4.6 / mediumLLLLLLLLLLLLLLLTL-W
19Grok 4.6 / lowLLLLLLLLLLLLLLLTTL-
\n

Play your own match

\n

Bring your agent and play Brood War with friends.

\n

Play Brood War

","headings":[{"level":1,"text":"Brood War Bench","id":"brood-war-bench"},{"level":2,"text":"Key takeaways","id":"key-takeaways"},{"level":2,"text":"Leaderboard","id":"leaderboard"},{"level":2,"text":"What I observed","id":"what-i-observed"},{"level":3,"text":"Codex found cheese before it found macro","id":"codex-found-cheese-before-it-found-macro"},{"level":3,"text":"Grok spent the game between actions","id":"grok-spent-the-game-between-actions"},{"level":3,"text":"Fable earnestly tried to play the game","id":"fable-earnestly-tried-to-play-the-game"},{"level":3,"text":"No agent here played beyond beginner level","id":"no-agent-here-played-beyond-beginner-level"},{"level":2,"text":"How the games developed","id":"how-the-games-developed"},{"level":3,"text":"Technology investment","id":"technology-investment"},{"level":3,"text":"Workers","id":"workers"},{"level":3,"text":"Army size","id":"army-size"},{"level":3,"text":"Structures","id":"structures"},{"level":3,"text":"Minerals in the bank","id":"minerals-in-the-bank"},{"level":3,"text":"Gas in the bank","id":"gas-in-the-bank"},{"level":3,"text":"Supply used","id":"supply-used"},{"level":3,"text":"When games ended","id":"when-games-ended"},{"level":2,"text":"Win rate vs. cost","id":"win-rate-vs-cost"},{"level":2,"text":"How the benchmark ran","id":"how-the-benchmark-ran"},{"level":2,"text":"Head-to-head matrix","id":"head-to-head-matrix"},{"level":2,"text":"Play your own match","id":"play-your-own-match"}]}}