{"article":{"slug":"memdoor-team-solve-five-qwen3-8-27b-models-match-a-research-harness-on-livecodebench-hard","title":"Memdoor team-solve: five Qwen3.8-27B models match a research harness on LiveCodeBench Hard","subtitle":null,"summary":"A Memdoor workflow write-up in which five small Qwen3.8-27B agents (planner, two ideators, solver, manager plus a repair worker) collaborate through files on disk, reproducing the GVS5H method and scoring 9.8/10 on ten LiveCodeBench Hard problems versus 8.0 for Fable 5 in one call, with honest caveats on sample size and contamination.","content_type":"research","language":"en","canonical_url":"https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve","author":{"name":"guregodevo","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Memdoor","url":"https://memdoor.ai/","listing_slug":null,"listing":null},"topics":[{"name":"ai","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"ai-agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"llm","slug":"llm","url":"https://listedarticles.com/topics/llm"},{"name":"benchmarks","slug":"benchmarks","url":"https://listedarticles.com/topics/benchmarks"},{"name":"open-source","slug":"open-source","url":"https://listedarticles.com/topics/open-source"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":652,"reading_minutes":3,"published_at":"2026-10-25T19:36:16.000Z","added_at":"2026-11-01T14:10:20.453Z","updated_at":"2026-11-01T14:10:20.453Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/memdoor-team-solve-five-qwen3-8-27b-models-match-a-research-harness-on-livecodebench-hard","markdown_url":"https://listedarticles.com/articles/memdoor-team-solve-five-qwen3-8-27b-models-match-a-research-harness-on-livecodebench-hard.md","example":false,"citation":"guregodevo, Memdoor. \"Memdoor team-solve: five Qwen3.8-27B models match a research harness on LiveCodeBench Hard.\" 25 Oct 2026. https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve"},"body_markdown":"Five small models solve a programming problem together: a planner, two\nideators, a solver and a manager, plus a repair worker. Each starts from a\nfresh context and talks only through files on disk. A script that runs the\nsamples decides success, not a model. It is the method of\n[GVS5H](https://github.com/slee-persis/GVS5H) (Lee, 2026), written as a\n[Memdoor](https://github.com/guregodevo/memdoor) workflow: eight YAML files, no code, no model\nnamed.\n\n## The result\n\nThe authors' harness rebuilt on the same model, Qwen3.8-27B, with their\nprompts, loop and 128K reply cap. The first ten problems of their\nLiveCodeBench Hard list, five passes each, graded on every hidden test with\ntheir corrected grader:\n\n| pass | this workflow | authors' harness | Fable 5, one call |\n|---|---|---|---|\n| 1 | **10** / 10 | 9 | 7 |\n| 2 | **10** / 10 | 10 | 8 |\n| 3 | **10** / 10 | 10 | 10 |\n| 4 | **10** / 10 | 9 | 8 |\n| 5 | **9** / 10 | 10 | 7 |\n| mean | **9.8** | 9.6 | 8.0 |\n\nThe authors' two columns are the regraded result files in their repository,\nthe numbers in their paper. Our column is\n[`bench/results/parity-5pass/`](https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve/bench/results/parity-5pass/): all 50 runs,\neach with every request and reply, the program, the grader's output and the\ncost. A run took a median of 40 minutes and $0.39; the 50 cost $35.79.\n\nWhat this does and does not show:\n\n- Ten problems, five passes, not the authors' 100. On a sample this small,\n  9.8 and 9.6 are the same number. The claim is parity with their harness, not\n  a gain on it.\n- The contests date from spring 2025 and the model may have seen them in\n  training. Contamination cannot be ruled out, for any column.\n- Fable 5's column is one call per problem, from the authors' files. It was\n  not run in this harness.\n- The loop sees only the public samples; the score is the hidden tests.\n- The model ran on OpenRouter, not a local machine.\n- The one lost problem: the manager invented a wrong counterexample and the\n  team rewrote a correct program. Twice elsewhere a correct program was\n  overwritten and then recovered. Both gaps are the authors' loop, kept as is.\n\nThe parity check (every request byte for byte against their code), the hosts,\nthe changes it took and the first run's findings are in [NOTES.md](https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/NOTES.md).\nThe same result, on one page: [memdoor.ai/docs/team-solve](https://memdoor.ai/docs/team-solve).\n\n## Run it\n\n    curl -fsSL https://memdoor.ai/install.sh | bash\n    memdoor connect openrouter        # or anthropic, openai, gemini, chatgpt\n    # or no key: memdoor connect local   (Ollama, llama.cpp, vLLM, LM Studio; Ollama needs a 32K window, not its 4K default)\n    git clone --depth 1 https://github.com/guregodevo/memdoor\n\n    mkdir -p ~/.memdoor/workflows\n    cp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/\n    mkdir my-problem && cd my-problem\n    cp -r ../memdoor/examples/workflows/team-solve/sample/* .    # task.md + tests/1.in … 5.out\n    memdoor workflow run team-solve --wait   # exit 0 when every sample passes\n\nFor your own problem, replace `task.md` and put paired `tests/N.in` /\n`tests/N.out` beside it. The program is `solution.py`, read from stdin,\ncompared to the samples with `diff`; it needs Python 3 and bash. `/workflow`\nin `memdoor tui` draws the run live.\n\nThe steps name no model: whatever `/model` is pinned to answers, a local one\nincluded ([how to connect one](https://memdoor.ai/docs/providers#a-local-runtime-no-key)).\nTo change a role, change its prompt in `steps/`.\n\n## How it works\n\n```\nplan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify\n       └── ideate-b ──┘\n```\n\n| step | kind | reads | writes |\n|---|---|---|---|\n| `plan` | agent | `task.md` | `plan.md`, `notes.md` |\n| `ideate-a`, `ideate-b` | agent, in parallel | the plan | a proposal each: the most direct provable algorithm, and one from another family |\n| `solve` | agent | both proposals | `solution.py`, the decision in `notes.md` |\n| `test` | command | `tests/` | `test-output.log` (never fails the run) |\n| `review` | agent | the test output | repair instructions in `notes.md` |\n| `repair` | agent | the instructions | a fixed `solution.py`, or nothing |\n| `verify` | command | `tests/` | the verdict: a nonzero exit fails the run |\n\nEvery agent prompt says \"fresh context, read the files, do not rely on chat\".\nThe paper credits that ledger discipline for the gain. Each step is one YAML\nfile in `steps/`, in the format of [mario](https://github.com/guregodevo/mario),\nthe engine underneath; `requires:` is the DAG.\n","body_html":"<p>Five small models solve a programming problem together: a planner, two\nideators, a solver and a manager, plus a repair worker. Each starts from a\nfresh context and talks only through files on disk. A script that runs the\nsamples decides success, not a model. It is the method of\n<a href=\"https://github.com/slee-persis/GVS5H\" rel=\"nofollow ugc noopener\">GVS5H</a> (Lee, 2026), written as a\n<a href=\"https://github.com/guregodevo/memdoor\" rel=\"nofollow ugc noopener\">Memdoor</a> workflow: eight YAML files, no code, no model\nnamed.</p>\n<h2 id=\"the-result\">The result</h2>\n<p>The authors&#39; harness rebuilt on the same model, Qwen3.8-27B, with their\nprompts, loop and 128K reply cap. The first ten problems of their\nLiveCodeBench Hard list, five passes each, graded on every hidden test with\ntheir corrected grader:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>pass</th><th>this workflow</th><th>authors&#39; harness</th><th>Fable 5, one call</th></tr></thead><tbody><tr><td>1</td><td><strong>10</strong> / 10</td><td>9</td><td>7</td></tr><tr><td>2</td><td><strong>10</strong> / 10</td><td>10</td><td>8</td></tr><tr><td>3</td><td><strong>10</strong> / 10</td><td>10</td><td>10</td></tr><tr><td>4</td><td><strong>10</strong> / 10</td><td>9</td><td>8</td></tr><tr><td>5</td><td>*<em>9*</em> / 10</td><td>10</td><td>7</td></tr><tr><td>mean</td><td><strong>9.8</strong></td><td>9.6</td><td>8.0</td></tr></tbody></table></div>\n<p>The authors&#39; two columns are the regraded result files in their repository,\nthe numbers in their paper. Our column is\n<a href=\"https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve/bench/results/parity-5pass/\" rel=\"nofollow ugc noopener\"><code>bench/results/parity-5pass/</code></a>: all 50 runs,\neach with every request and reply, the program, the grader&#39;s output and the\ncost. A run took a median of 40 minutes and $0.39; the 50 cost $35.79.</p>\n<p>What this does and does not show:</p>\n<ul><li><p>Ten problems, five passes, not the authors&#39; 100. On a sample this small,</p><p>9.8 and 9.6 are the same number. The claim is parity with their harness, not\na gain on it.</p></li><li><p>The contests date from spring 2025 and the model may have seen them in</p><p>training. Contamination cannot be ruled out, for any column.</p></li><li><p>Fable 5&#39;s column is one call per problem, from the authors&#39; files. It was</p><p>not run in this harness.</p></li><li>The loop sees only the public samples; the score is the hidden tests.</li><li>The model ran on OpenRouter, not a local machine.</li><li><p>The one lost problem: the manager invented a wrong counterexample and the</p><p>team rewrote a correct program. Twice elsewhere a correct program was\noverwritten and then recovered. Both gaps are the authors&#39; loop, kept as is.</p></li></ul>\n<p>The parity check (every request byte for byte against their code), the hosts,\nthe changes it took and the first run&#39;s findings are in <a href=\"https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/NOTES.md\" rel=\"nofollow ugc noopener\">NOTES.md</a>.\nThe same result, on one page: <a href=\"https://memdoor.ai/docs/team-solve\" rel=\"nofollow ugc noopener\">memdoor.ai/docs/team-solve</a>.</p>\n<h2 id=\"run-it\">Run it</h2>\n<pre><code>curl -fsSL https://memdoor.ai/install.sh | bash\nmemdoor connect openrouter        # or anthropic, openai, gemini, chatgpt\n# or no key: memdoor connect local   (Ollama, llama.cpp, vLLM, LM Studio; Ollama needs a 32K window, not its 4K default)\ngit clone --depth 1 https://github.com/guregodevo/memdoor\n\nmkdir -p ~/.memdoor/workflows\ncp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/\nmkdir my-problem &amp;&amp; cd my-problem\ncp -r ../memdoor/examples/workflows/team-solve/sample/* .    # task.md + tests/1.in … 5.out\nmemdoor workflow run team-solve --wait   # exit 0 when every sample passes</code></pre>\n<p>For your own problem, replace <code>task.md</code> and put paired <code>tests/N.in</code> /\n<code>tests/N.out</code> beside it. The program is <code>solution.py</code>, read from stdin,\ncompared to the samples with <code>diff</code>; it needs Python 3 and bash. <code>/workflow</code>\nin <code>memdoor tui</code> draws the run live.</p>\n<p>The steps name no model: whatever <code>/model</code> is pinned to answers, a local one\nincluded (<a href=\"https://memdoor.ai/docs/providers#a-local-runtime-no-key\" rel=\"nofollow ugc noopener\">how to connect one</a>).\nTo change a role, change its prompt in <code>steps/</code>.</p>\n<h2 id=\"how-it-works\">How it works</h2>\n<pre><code>plan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify\n       └── ideate-b ──┘</code></pre>\n<div class=\"table-wrap\"><table><thead><tr><th>step</th><th>kind</th><th>reads</th><th>writes</th></tr></thead><tbody><tr><td><code>plan</code></td><td>agent</td><td><code>task.md</code></td><td><code>plan.md</code>, <code>notes.md</code></td></tr><tr><td><code>ideate-a</code>, <code>ideate-b</code></td><td>agent, in parallel</td><td>the plan</td><td>a proposal each: the most direct provable algorithm, and one from another family</td></tr><tr><td><code>solve</code></td><td>agent</td><td>both proposals</td><td><code>solution.py</code>, the decision in <code>notes.md</code></td></tr><tr><td><code>test</code></td><td>command</td><td><code>tests/</code></td><td><code>test-output.log</code> (never fails the run)</td></tr><tr><td><code>review</code></td><td>agent</td><td>the test output</td><td>repair instructions in <code>notes.md</code></td></tr><tr><td><code>repair</code></td><td>agent</td><td>the instructions</td><td>a fixed <code>solution.py</code>, or nothing</td></tr><tr><td><code>verify</code></td><td>command</td><td><code>tests/</code></td><td>the verdict: a nonzero exit fails the run</td></tr></tbody></table></div>\n<p>Every agent prompt says &quot;fresh context, read the files, do not rely on chat&quot;.\nThe paper credits that ledger discipline for the gain. Each step is one YAML\nfile in <code>steps/</code>, in the format of <a href=\"https://github.com/guregodevo/mario\" rel=\"nofollow ugc noopener\">mario</a>,\nthe engine underneath; <code>requires:</code> is the DAG.</p>","headings":[{"level":2,"text":"The result","id":"the-result"},{"level":2,"text":"Run it","id":"run-it"},{"level":2,"text":"How it works","id":"how-it-works"}]}}