{"article":{"slug":"frontierphysics-evaluating-agents-for-end-to-end-physics-research","title":"FrontierPhysics: Evaluating Agents for End-to-End Physics Research","subtitle":null,"summary":"BenchFlow’s FrontierPhysics v0.1 benchmarks AI agents on 56 end-to-end physics research tasks written by 42 physicists; frontier models struggle most on rigorous reasoning and expert intuition.","content_type":"research","language":"en","canonical_url":"https://www.benchflow.ai/frontierphysics","author":{"name":null,"url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"BenchFlow","url":"https://www.benchflow.ai/","listing_slug":null,"listing":null},"topics":[{"name":"AI","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"Benchmarks","slug":"benchmarks","url":"https://listedarticles.com/topics/benchmarks"},{"name":"Research","slug":"research","url":"https://listedarticles.com/topics/research"},{"name":"Science","slug":"science","url":"https://listedarticles.com/topics/science"},{"name":"AI Agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":838,"reading_minutes":4,"published_at":"2026-11-05T00:00:00.000Z","added_at":"2026-11-06T08:23:06.929Z","updated_at":"2026-11-06T08:23:06.929Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/frontierphysics-evaluating-agents-for-end-to-end-physics-research","markdown_url":"https://listedarticles.com/articles/frontierphysics-evaluating-agents-for-end-to-end-physics-research.md","example":false,"citation":"BenchFlow. \"FrontierPhysics: Evaluating Agents for End-to-End Physics Research.\" 5 Nov 2026. https://www.benchflow.ai/frontierphysics (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://www.benchflow.ai/frontierphysics"},"body_markdown":"[FrontierPhysics v0.1 with 56 tasks](https://arxiv.org/abs/2611.06910)\n\n# FrontierPhysics: Evaluating Agents for End-to-End Physics Research\n\n182submissions·56tasks·9domains·42researchers·26institutions\n\n[Read the Paper](https://arxiv.org/abs/2611.06910)[Contribute a Task](/frontierphysics/contribute)[Benchmark Collaboration](mailto:bingran@benchflow.ai,xiangyi@benchflow.ai?subject=FrontierPhysics%20benchmark%20collaboration)\n\n- ![BenchFlow](/frontierphysics/logos/benchflow.png \"BenchFlow\")\n- ![University of California, Berkeley](/frontierphysics/logos/uc-berkeley.svg \"University of California, Berkeley\")\n- ![California Institute of Technology](/frontierphysics/logos/caltech.svg \"California Institute of Technology\")\n- ![Harvard University](/frontierphysics/logos/harvard.svg \"Harvard University\")\n- ![University of Washington](/frontierphysics/logos/uw.svg \"University of Washington\")\n- ![Stanford University](/frontierphysics/logos/stanford.svg \"Stanford University\")\n- ![Amazon](/frontierphysics/logos/amazon.svg \"Amazon\")\n- ![University of Pittsburgh](/frontierphysics/logos/pitt.svg \"University of Pittsburgh\")\n- ![University of California, San Diego](/frontierphysics/logos/ucsd.svg \"University of California, San Diego\")\n- ![ZEISS](/frontierphysics/logos/zeiss.svg \"ZEISS\")\n- ![Georgia Institute of Technology](/frontierphysics/logos/georgia-tech.svg \"Georgia Institute of Technology\")\n- ![Lawrence Berkeley National Laboratory](/frontierphysics/logos/berkeley-lab.svg \"Lawrence Berkeley National Laboratory\")\n- ![University of Michigan, Ann Arbor](/frontierphysics/logos/umich.svg \"University of Michigan, Ann Arbor\")\n- ![The Scripps Research Institute](/frontierphysics/logos/scripps.svg \"The Scripps Research Institute\")\n- ![Oak Ridge National Laboratory](/frontierphysics/logos/ornl.svg \"Oak Ridge National Laboratory\")\n- ![University of Texas at Austin](/frontierphysics/logos/ut-austin.svg \"University of Texas at Austin\")\n- ![Purdue University](/frontierphysics/logos/purdue.svg \"Purdue University\")\n- ![University of Toronto](/frontierphysics/logos/toronto.png \"University of Toronto\")\n- ![University of Southern California](/frontierphysics/logos/usc.svg \"University of Southern California\")\n- ![Ludwig-Maximilians-Universität München](/frontierphysics/logos/lmu.svg \"Ludwig-Maximilians-Universität München\")\n- ![Carnegie Mellon University](/frontierphysics/logos/cmu.svg \"Carnegie Mellon University\")\n- ![University of California, Davis](/frontierphysics/logos/ucdavis.svg \"University of California, Davis\")\n- ![Princeton University](/frontierphysics/logos/princeton.svg \"Princeton University\")\n- ![Northeastern University](/frontierphysics/logos/northeastern.png \"Northeastern University\")\n- ![University of Pennsylvania](/frontierphysics/logos/upenn.svg \"University of Pennsylvania\")\n- ![University of Minnesota, Twin Cities](/frontierphysics/logos/umn.svg \"University of Minnesota, Twin Cities\")\n- ![New York University](/frontierphysics/logos/nyu.svg \"New York University\")\n- ![Duke University](/frontierphysics/logos/duke.png \"Duke University\")\n- ![Nanyang Technological University](/frontierphysics/logos/ntu.svg \"Nanyang Technological University\")\n- ![Pipeline Lab](/frontierphysics/logos/pipeline-lab.png \"Pipeline Lab\")\n- ![RLWRLD](/frontierphysics/logos/rlwrld.svg \"RLWRLD\")\n- ![University of Palermo](/frontierphysics/logos/unipa.svg \"University of Palermo\")\n- ![IBM](/frontierphysics/logos/ibm.svg \"IBM\")\n- ![University of Amsterdam](/frontierphysics/logos/uva.svg \"University of Amsterdam\")\n- ![University of California, Santa Cruz](/frontierphysics/logos/ucsc.svg \"University of California, Santa Cruz\")\n- ![Prime Intellect](/frontierphysics/logos/prime-intellect.svg \"Prime Intellect\")\n\n- ![BenchFlow](/frontierphysics/logos/benchflow.png \"BenchFlow\")\n- ![University of California, Berkeley](/frontierphysics/logos/uc-berkeley.svg \"University of California, Berkeley\")\n- ![California Institute of Technology](/frontierphysics/logos/caltech.svg \"California Institute of Technology\")\n- ![Harvard University](/frontierphysics/logos/harvard.svg \"Harvard University\")\n- ![University of Washington](/frontierphysics/logos/uw.svg \"University of Washington\")\n- ![Stanford University](/frontierphysics/logos/stanford.svg \"Stanford University\")\n- ![Amazon](/frontierphysics/logos/amazon.svg \"Amazon\")\n- ![University of Pittsburgh](/frontierphysics/logos/pitt.svg \"University of Pittsburgh\")\n- ![University of California, San Diego](/frontierphysics/logos/ucsd.svg \"University of California, San Diego\")\n- ![ZEISS](/frontierphysics/logos/zeiss.svg \"ZEISS\")\n- ![Georgia Institute of Technology](/frontierphysics/logos/georgia-tech.svg \"Georgia Institute of Technology\")\n- ![Lawrence Berkeley National Laboratory](/frontierphysics/logos/berkeley-lab.svg \"Lawrence Berkeley National Laboratory\")\n- ![University of Michigan, Ann Arbor](/frontierphysics/logos/umich.svg \"University of Michigan, Ann Arbor\")\n- ![The Scripps Research Institute](/frontierphysics/logos/scripps.svg \"The Scripps Research Institute\")\n- ![Oak Ridge National Laboratory](/frontierphysics/logos/ornl.svg \"Oak Ridge National Laboratory\")\n- ![University of Texas at Austin](/frontierphysics/logos/ut-austin.svg \"University of Texas at Austin\")\n- ![Purdue University](/frontierphysics/logos/purdue.svg \"Purdue University\")\n- ![University of Toronto](/frontierphysics/logos/toronto.png \"University of Toronto\")\n- ![University of Southern California](/frontierphysics/logos/usc.svg \"University of Southern California\")\n- ![Ludwig-Maximilians-Universität München](/frontierphysics/logos/lmu.svg \"Ludwig-Maximilians-Universität München\")\n- ![Carnegie Mellon University](/frontierphysics/logos/cmu.svg \"Carnegie Mellon University\")\n- ![University of California, Davis](/frontierphysics/logos/ucdavis.svg \"University of California, Davis\")\n- ![Princeton University](/frontierphysics/logos/princeton.svg \"Princeton University\")\n- ![Northeastern University](/frontierphysics/logos/northeastern.png \"Northeastern University\")\n- ![University of Pennsylvania](/frontierphysics/logos/upenn.svg \"University of Pennsylvania\")\n- ![University of Minnesota, Twin Cities](/frontierphysics/logos/umn.svg \"University of Minnesota, Twin Cities\")\n- ![New York University](/frontierphysics/logos/nyu.svg \"New York University\")\n- ![Duke University](/frontierphysics/logos/duke.png \"Duke University\")\n- ![Nanyang Technological University](/frontierphysics/logos/ntu.svg \"Nanyang Technological University\")\n- ![Pipeline Lab](/frontierphysics/logos/pipeline-lab.png \"Pipeline Lab\")\n- ![RLWRLD](/frontierphysics/logos/rlwrld.svg \"RLWRLD\")\n- ![University of Palermo](/frontierphysics/logos/unipa.svg \"University of Palermo\")\n- ![IBM](/frontierphysics/logos/ibm.svg \"IBM\")\n- ![University of Amsterdam](/frontierphysics/logos/uva.svg \"University of Amsterdam\")\n- ![University of California, Santa Cruz](/frontierphysics/logos/ucsc.svg \"University of California, Santa Cruz\")\n- ![Prime Intellect](/frontierphysics/logos/prime-intellect.svg \"Prime Intellect\")\n\n## How It Works\n\n1. [1\n\n   ### Build tasks\n\n   182 submissions\n\n   - 42 physicists\n   - 26 institutions\n\n   182 submissions](#tasks)\n2. [2\n\n   ### Quality review\n\n   56 tasks · 9 domains\n\n   - General quality review\n   - Domain peer review\n\n   56 tasks · 9 domains](#tasks)\n3. [3\n\n   ### Run AI agents\n\n   4,364 rollouts\n\n   - 22 models\n   - 4 harnesses\n\n   4,364 rollouts](#performance)\n4. [4\n\n   ### Grade rollouts\n\n   Score 0–100\n\n   - Verifier\n   - Rubrics\n\n   Score 0–100](#leaderboard)\n5. [5\n\n   ### Failure-mode analysis\n\n   14 modes\n\n   - RIG48%\n   - INT34%\n   - LIT27%\n   - INS26%\n   - FALSE7%\n   - RRMIS3%\n\n   14 modes](#failure-modes)\n\n## Agent Performance\n\n## Failure Modes\n\nGPT-6 Astra (xhigh)\n\nOne square per rollout · 56 tasks × 3 trialsRIG68 of 168\n\nCondensed Matter\n\nAMO\n\nQuantum Info\n\nAstrophysics\n\nFluids\n\nStatistical\n\nPhotonics\n\nHigh Energy\n\nChemical\n\nAgent failuresn · % of 143 failed\n\nGrading errors\n\nAlso in the taxonomy\n\n## Agent Leaderboard\n\nSort by\n\n| # | Agent | Verifier | Overall | Pass@3 | Score | Visualization |\n| --- | --- | --- | --- | --- | --- | --- |\n| 1 | Claude Opus 5.5Claude Code | 45.8% | 27.4% | 44.6% | 26.0 | 45.8 → 27.4% |\n| 2 | GPT-6 Astra (max)Codex | 39.3% | 23.2% | 30.4% | 21.8 | 39.3 → 23.2% |\n| 3 | GPT-6 Astra (xhigh)Codex | 38.7% | 19.6% | 30.4% | 18.0 | 38.7 → 19.6% |\n| 4 | GPT-6.1 SolCodex | 33.3% | 19.6% | 30.4% | 18.4 | 33.3 → 19.6% |\n| 5 | GPT-6.1 TerraCodex | 28.6% | 14.3% | 21.4% | 12.6 | 28.6 → 14.3% |\n| 6 | GPT-6 SolCodex | 27.4% | 14.3% | 28.6% | 12.5 | 27.4 → 14.3% |\n| 7 | Claude Fable 5.1Claude Code | 31.0% | 11.3% | 21.4% | 10.2 | 31.0 → 11.3% |\n| 8 | Claude Opus 5Claude Code | 20.2% | 9.5% | 17.9% | 8.6 | 20.2 → 9.5% |\n| 9 | Gemini 4 ArgonAntigravity | 31.5% | 8.3% | 17.9% | 6.5 | 31.5 → 8.3% |\n| 10 | GPT-5.6 Sol (max)Codex | 19.6% | 7.1% | 12.5% | 6.2 | 19.6 → 7.1% |\n| 11 | Kimi K3.1 ProClaude Code | 15.5% | 6.5% | 12.5% | 6.1 | 15.5 → 6.5% |\n| 12 | Grok 4.7Grok Build | 16.7% | 4.2% | 10.7% | 3.8 | 16.7 → 4.2% |\n| 13 | Claude Haiku 5.5Claude Code | 15.5% | 4.2% | 10.7% | 3.6 | 15.5 → 4.2% |\n| 14 | GPT-5.6 Sol (xhigh)Codex | 15.5% | 4.2% | 8.9% | 3.5 | 15.5 → 4.2% |\n| 15 | GLM-5.3Claude Code | 7.1% | 4.2% | 8.9% | 3.6 | 7.1 → 4.2% |\n| 16 | GPT-5.6 Terra (max)Codex | 16.1% | 3.6% | 7.1% | 2.9 | 16.1 → 3.6% |\n| 17 | GPT-5.6 Luna (max)Codex | 12.5% | 3.6% | 3.6% | 2.9 | 12.5 → 3.6% |\n| 18 | DeepSeek V4.1 FlashClaude Code | 10.7% | 3.6% | 5.4% | 3.3 | 10.7 → 3.6% |\n| 19 | Kimi K3Claude Code | 11.3% | 2.4% | 5.4% | 1.8 | 11.3 → 2.4% |\n| 20 | Grok 4.6Grok Build | 10.7% | 2.4% | 3.6% | 2.2 | 10.7 → 2.4% |\n| 21 | GPT-5.6 Luna (xhigh)Codex | 8.9% | 1.8% | 3.6% | 1.4 | 8.9 → 1.8% |\n| 22 | MiMo V2.6 ProClaude Code | 10.1% | 1.2% | 3.6% | 1.0 | 10.1 → 1.2% |\n| 23 | GPT-5.6 Terra (xhigh)Codex | 9.5% | 1.2% | 1.8% | 0.9 | 9.5 → 1.2% |\n| 24 | DeepSeek V4 ProClaude Code | 3.0% | 0.6% | 1.8% | 0.3 | 3.0 → 0.6% |\n| 25 | Gemini 3.8 FlashAntigravity | 7.7% | 0.0% | 0.0% | 0.0 | 7.7 → 0.0% |\n| 26 | Mistral Large 4Claude Code | 5.4% | 0.0% | 0.0% | 0.0 | 5.4 → 0.0% |\n\nFrontierPhysics · 22 models · 26 configurations · 56 tasks · 3 trials per task · 95% CIsOverallVerifier only\n\nAnthropic\n\nOpenAI\n\nGoogle\n\nMoonshot\n\nxAI\n\nZ.ai\n\nDeepSeek\n\nXiaomi\n\nMistral\n\n## Cost Efficiency\n\n## Physics-Domain Profile\n\nCondensed Matter Physics12 tasks\n\nClaude Opus 5.5Claude Code44.4 → 33.3%\n\nGPT-6 Astra (max)Codex30.6 → 25.0%\n\nGemini 4 ArgonAntigravity47.2 → 13.9%\n\nsolid = overall pass · pale = passed the verifier only · hover another radar axis to switch domain\n\nClaude Opus 5.5· Claude CodeGPT-6 Astra (max)· CodexGemini 4 Argon· Antigravityrings at 20–80% · overall pass\n\n## Tasks\n\ndays per original project, on average\n:   ~270\n\nhours of research in total\n:   8,000+\n\ntasks from the contributors' published papers\n:   18\n\ncitations per task author, on average\n:   420\n\n![A figure from a FrontierPhysics task in Condensed Matter Physics](/frontierphysics/task-gallery/fig-01.webp)\n\nCondensed Matter Physics\n\n![A figure from a FrontierPhysics task in AMO Physics](/frontierphysics/task-gallery/fig-13.webp)\n\nAMO Physics\n\n![A figure from a FrontierPhysics task in Quantum Information](/frontierphysics/task-gallery/fig-20.webp)\n\nQuantum Information\n\n![A figure from a FrontierPhysics task in Astrophysics](/frontierphysics/task-gallery/fig-25.webp)\n\nAstrophysics\n\n![A figure from a FrontierPhysics task in Fluid Dynamics](/frontierphysics/task-gallery/fig-31.webp)\n\nFluid Dynamics\n\n![A figure from a FrontierPhysics task in Statistical Physics](/frontierphysics/task-gallery/fig-37.webp)\n\nStatistical Physics\n\n[Figure credits](/frontierphysics/credits)\n\n## Example Task\n\n### rydberg-ising-critical-point\n\nhardatomic-molecular-and-optical-physicssimulationcalculationanalysis\n\ntasks/rydberg-ising-critical-point/task.md\n\n- - device7 files\n  - paper\\_template16 files\n- oracle2 files\n\nLoading…\n\n### Run eval\n\nclaude-agent-acp · effort max · no-skill · trial 3 of 3\n\n[Contribute a Task](/frontierphysics/contribute)\n","body_html":"<p><a href=\"https://arxiv.org/abs/2611.06910\" rel=\"nofollow ugc noopener\">FrontierPhysics v0.1 with 56 tasks</a></p>\n<h1 id=\"frontierphysics-evaluating-agents-for-end-to-end-physics-researc\">FrontierPhysics: Evaluating Agents for End-to-End Physics Research</h1>\n<p>182submissions·56tasks·9domains·42researchers·26institutions</p>\n<p><a href=\"https://arxiv.org/abs/2611.06910\" rel=\"nofollow ugc noopener\">Read the Paper</a><a href=\"/frontierphysics/contribute\">Contribute a Task</a><a href=\"mailto:bingran@benchflow.ai,xiangyi@benchflow.ai?subject=FrontierPhysics%20benchmark%20collaboration\">Benchmark Collaboration</a></p>\n<ul><li>BenchFlow</li><li>University of California, Berkeley</li><li>California Institute of Technology</li><li>Harvard University</li><li>University of Washington</li><li>Stanford University</li><li>Amazon</li><li>University of Pittsburgh</li><li>University of California, San Diego</li><li>ZEISS</li><li>Georgia Institute of Technology</li><li>Lawrence Berkeley National Laboratory</li><li>University of Michigan, Ann Arbor</li><li>The Scripps Research Institute</li><li>Oak Ridge National Laboratory</li><li>University of Texas at Austin</li><li>Purdue University</li><li>University of Toronto</li><li>University of Southern California</li><li>Ludwig-Maximilians-Universität München</li><li>Carnegie Mellon University</li><li>University of California, Davis</li><li>Princeton University</li><li>Northeastern University</li><li>University of Pennsylvania</li><li>University of Minnesota, Twin Cities</li><li>New York University</li><li>Duke University</li><li>Nanyang Technological University</li><li>Pipeline Lab</li><li>RLWRLD</li><li>University of Palermo</li><li>IBM</li><li>University of Amsterdam</li><li>University of California, Santa Cruz</li><li>Prime Intellect</li><li>BenchFlow</li><li>University of California, Berkeley</li><li>California Institute of Technology</li><li>Harvard University</li><li>University of Washington</li><li>Stanford University</li><li>Amazon</li><li>University of Pittsburgh</li><li>University of California, San Diego</li><li>ZEISS</li><li>Georgia Institute of Technology</li><li>Lawrence Berkeley National Laboratory</li><li>University of Michigan, Ann Arbor</li><li>The Scripps Research Institute</li><li>Oak Ridge National Laboratory</li><li>University of Texas at Austin</li><li>Purdue University</li><li>University of Toronto</li><li>University of Southern California</li><li>Ludwig-Maximilians-Universität München</li><li>Carnegie Mellon University</li><li>University of California, Davis</li><li>Princeton University</li><li>Northeastern University</li><li>University of Pennsylvania</li><li>University of Minnesota, Twin Cities</li><li>New York University</li><li>Duke University</li><li>Nanyang Technological University</li><li>Pipeline Lab</li><li>RLWRLD</li><li>University of Palermo</li><li>IBM</li><li>University of Amsterdam</li><li>University of California, Santa Cruz</li><li>Prime Intellect</li></ul>\n<h2 id=\"how-it-works\">How It Works</h2>\n<ol><li><p>[1</p><p> ### Build tasks</p>\n<p> 182 submissions</p>\n<ul><li>42 physicists</li><li>26 institutions</li></ul>\n<p> 182 submissions](#tasks)</p></li><li><p>[2</p><p> ### Quality review</p>\n<p> 56 tasks · 9 domains</p>\n<ul><li>General quality review</li><li>Domain peer review</li></ul>\n<p> 56 tasks · 9 domains](#tasks)</p></li><li><p>[3</p><p> ### Run AI agents</p>\n<p> 4,364 rollouts</p>\n<ul><li>22 models</li><li>4 harnesses</li></ul>\n<p> 4,364 rollouts](#performance)</p></li><li><p>[4</p><p> ### Grade rollouts</p>\n<p> Score 0–100</p>\n<ul><li>Verifier</li><li>Rubrics</li></ul>\n<p> Score 0–100](#leaderboard)</p></li><li><p>[5</p><p> ### Failure-mode analysis</p>\n<p> 14 modes</p>\n<ul><li>RIG48%</li><li>INT34%</li><li>LIT27%</li><li>INS26%</li><li>FALSE7%</li><li>RRMIS3%</li></ul>\n<p> 14 modes](#failure-modes)</p></li></ol>\n<h2 id=\"agent-performance\">Agent Performance</h2>\n<h2 id=\"failure-modes\">Failure Modes</h2>\n<p>GPT-6 Astra (xhigh)</p>\n<p>One square per rollout · 56 tasks × 3 trialsRIG68 of 168</p>\n<p>Condensed Matter</p>\n<p>AMO</p>\n<p>Quantum Info</p>\n<p>Astrophysics</p>\n<p>Fluids</p>\n<p>Statistical</p>\n<p>Photonics</p>\n<p>High Energy</p>\n<p>Chemical</p>\n<p>Agent failuresn · % of 143 failed</p>\n<p>Grading errors</p>\n<p>Also in the taxonomy</p>\n<h2 id=\"agent-leaderboard\">Agent Leaderboard</h2>\n<p>Sort by</p>\n<div class=\"table-wrap\"><table><thead><tr><th>#</th><th>Agent</th><th>Verifier</th><th>Overall</th><th>Pass@3</th><th>Score</th><th>Visualization</th></tr></thead><tbody><tr><td>1</td><td>Claude Opus 5.5Claude Code</td><td>45.8%</td><td>27.4%</td><td>44.6%</td><td>26.0</td><td>45.8 → 27.4%</td></tr><tr><td>2</td><td>GPT-6 Astra (max)Codex</td><td>39.3%</td><td>23.2%</td><td>30.4%</td><td>21.8</td><td>39.3 → 23.2%</td></tr><tr><td>3</td><td>GPT-6 Astra (xhigh)Codex</td><td>38.7%</td><td>19.6%</td><td>30.4%</td><td>18.0</td><td>38.7 → 19.6%</td></tr><tr><td>4</td><td>GPT-6.1 SolCodex</td><td>33.3%</td><td>19.6%</td><td>30.4%</td><td>18.4</td><td>33.3 → 19.6%</td></tr><tr><td>5</td><td>GPT-6.1 TerraCodex</td><td>28.6%</td><td>14.3%</td><td>21.4%</td><td>12.6</td><td>28.6 → 14.3%</td></tr><tr><td>6</td><td>GPT-6 SolCodex</td><td>27.4%</td><td>14.3%</td><td>28.6%</td><td>12.5</td><td>27.4 → 14.3%</td></tr><tr><td>7</td><td>Claude Fable 5.1Claude Code</td><td>31.0%</td><td>11.3%</td><td>21.4%</td><td>10.2</td><td>31.0 → 11.3%</td></tr><tr><td>8</td><td>Claude Opus 5Claude Code</td><td>20.2%</td><td>9.5%</td><td>17.9%</td><td>8.6</td><td>20.2 → 9.5%</td></tr><tr><td>9</td><td>Gemini 4 ArgonAntigravity</td><td>31.5%</td><td>8.3%</td><td>17.9%</td><td>6.5</td><td>31.5 → 8.3%</td></tr><tr><td>10</td><td>GPT-5.6 Sol (max)Codex</td><td>19.6%</td><td>7.1%</td><td>12.5%</td><td>6.2</td><td>19.6 → 7.1%</td></tr><tr><td>11</td><td>Kimi K3.1 ProClaude Code</td><td>15.5%</td><td>6.5%</td><td>12.5%</td><td>6.1</td><td>15.5 → 6.5%</td></tr><tr><td>12</td><td>Grok 4.7Grok Build</td><td>16.7%</td><td>4.2%</td><td>10.7%</td><td>3.8</td><td>16.7 → 4.2%</td></tr><tr><td>13</td><td>Claude Haiku 5.5Claude Code</td><td>15.5%</td><td>4.2%</td><td>10.7%</td><td>3.6</td><td>15.5 → 4.2%</td></tr><tr><td>14</td><td>GPT-5.6 Sol (xhigh)Codex</td><td>15.5%</td><td>4.2%</td><td>8.9%</td><td>3.5</td><td>15.5 → 4.2%</td></tr><tr><td>15</td><td>GLM-5.3Claude Code</td><td>7.1%</td><td>4.2%</td><td>8.9%</td><td>3.6</td><td>7.1 → 4.2%</td></tr><tr><td>16</td><td>GPT-5.6 Terra (max)Codex</td><td>16.1%</td><td>3.6%</td><td>7.1%</td><td>2.9</td><td>16.1 → 3.6%</td></tr><tr><td>17</td><td>GPT-5.6 Luna (max)Codex</td><td>12.5%</td><td>3.6%</td><td>3.6%</td><td>2.9</td><td>12.5 → 3.6%</td></tr><tr><td>18</td><td>DeepSeek V4.1 FlashClaude Code</td><td>10.7%</td><td>3.6%</td><td>5.4%</td><td>3.3</td><td>10.7 → 3.6%</td></tr><tr><td>19</td><td>Kimi K3Claude Code</td><td>11.3%</td><td>2.4%</td><td>5.4%</td><td>1.8</td><td>11.3 → 2.4%</td></tr><tr><td>20</td><td>Grok 4.6Grok Build</td><td>10.7%</td><td>2.4%</td><td>3.6%</td><td>2.2</td><td>10.7 → 2.4%</td></tr><tr><td>21</td><td>GPT-5.6 Luna (xhigh)Codex</td><td>8.9%</td><td>1.8%</td><td>3.6%</td><td>1.4</td><td>8.9 → 1.8%</td></tr><tr><td>22</td><td>MiMo V2.6 ProClaude Code</td><td>10.1%</td><td>1.2%</td><td>3.6%</td><td>1.0</td><td>10.1 → 1.2%</td></tr><tr><td>23</td><td>GPT-5.6 Terra (xhigh)Codex</td><td>9.5%</td><td>1.2%</td><td>1.8%</td><td>0.9</td><td>9.5 → 1.2%</td></tr><tr><td>24</td><td>DeepSeek V4 ProClaude Code</td><td>3.0%</td><td>0.6%</td><td>1.8%</td><td>0.3</td><td>3.0 → 0.6%</td></tr><tr><td>25</td><td>Gemini 3.8 FlashAntigravity</td><td>7.7%</td><td>0.0%</td><td>0.0%</td><td>0.0</td><td>7.7 → 0.0%</td></tr><tr><td>26</td><td>Mistral Large 4Claude Code</td><td>5.4%</td><td>0.0%</td><td>0.0%</td><td>0.0</td><td>5.4 → 0.0%</td></tr></tbody></table></div>\n<p>FrontierPhysics · 22 models · 26 configurations · 56 tasks · 3 trials per task · 95% CIsOverallVerifier only</p>\n<p>Anthropic</p>\n<p>OpenAI</p>\n<p>Google</p>\n<p>Moonshot</p>\n<p>xAI</p>\n<p>Z.ai</p>\n<p>DeepSeek</p>\n<p>Xiaomi</p>\n<p>Mistral</p>\n<h2 id=\"cost-efficiency\">Cost Efficiency</h2>\n<h2 id=\"physics-domain-profile\">Physics-Domain Profile</h2>\n<p>Condensed Matter Physics12 tasks</p>\n<p>Claude Opus 5.5Claude Code44.4 → 33.3%</p>\n<p>GPT-6 Astra (max)Codex30.6 → 25.0%</p>\n<p>Gemini 4 ArgonAntigravity47.2 → 13.9%</p>\n<p>solid = overall pass · pale = passed the verifier only · hover another radar axis to switch domain</p>\n<p>Claude Opus 5.5· Claude CodeGPT-6 Astra (max)· CodexGemini 4 Argon· Antigravityrings at 20–80% · overall pass</p>\n<h2 id=\"tasks\">Tasks</h2>\n<p>days per original project, on average\n:   ~270</p>\n<p>hours of research in total\n:   8,000+</p>\n<p>tasks from the contributors&#39; published papers\n:   18</p>\n<p>citations per task author, on average\n:   420</p>\n<p>A figure from a FrontierPhysics task in Condensed Matter Physics</p>\n<p>Condensed Matter Physics</p>\n<p>A figure from a FrontierPhysics task in AMO Physics</p>\n<p>AMO Physics</p>\n<p>A figure from a FrontierPhysics task in Quantum Information</p>\n<p>Quantum Information</p>\n<p>A figure from a FrontierPhysics task in Astrophysics</p>\n<p>Astrophysics</p>\n<p>A figure from a FrontierPhysics task in Fluid Dynamics</p>\n<p>Fluid Dynamics</p>\n<p>A figure from a FrontierPhysics task in Statistical Physics</p>\n<p>Statistical Physics</p>\n<p><a href=\"/frontierphysics/credits\">Figure credits</a></p>\n<h2 id=\"example-task\">Example Task</h2>\n<h3 id=\"rydberg-ising-critical-point\">rydberg-ising-critical-point</h3>\n<p>hardatomic-molecular-and-optical-physicssimulationcalculationanalysis</p>\n<p>tasks/rydberg-ising-critical-point/task.md</p>\n<ul><li>- device7 files<ul><li>paper_template16 files</li></ul></li><li>oracle2 files</li></ul>\n<p>Loading…</p>\n<h3 id=\"run-eval\">Run eval</h3>\n<p>claude-agent-acp · effort max · no-skill · trial 3 of 3</p>\n<p><a href=\"/frontierphysics/contribute\">Contribute a Task</a></p>","headings":[{"level":1,"text":"FrontierPhysics: Evaluating Agents for End-to-End Physics Research","id":"frontierphysics-evaluating-agents-for-end-to-end-physics-researc"},{"level":2,"text":"How It Works","id":"how-it-works"},{"level":2,"text":"Agent Performance","id":"agent-performance"},{"level":2,"text":"Failure Modes","id":"failure-modes"},{"level":2,"text":"Agent Leaderboard","id":"agent-leaderboard"},{"level":2,"text":"Cost Efficiency","id":"cost-efficiency"},{"level":2,"text":"Physics-Domain Profile","id":"physics-domain-profile"},{"level":2,"text":"Tasks","id":"tasks"},{"level":2,"text":"Example Task","id":"example-task"},{"level":3,"text":"rydberg-ising-critical-point","id":"rydberg-ising-critical-point"},{"level":3,"text":"Run eval","id":"run-eval"}]}}