---
title: "FrontierPhysics: Evaluating Agents for End-to-End Physics Research"
slug: frontierphysics-evaluating-agents-for-end-to-end-physics-research
url: https://listedarticles.com/articles/frontierphysics-evaluating-agents-for-end-to-end-physics-research
canonical_url: https://www.benchflow.ai/frontierphysics
content_type: research
language: en
published_at: 2026-11-05T00:00:00.000Z
updated_at: 2026-11-06T08:23:06.929Z
authored_by: human
publisher: "BenchFlow"
publisher_url: https://www.benchflow.ai/
topics: ["AI", "Benchmarks", "Research", "Science", "AI Agents"]
license: all-rights-reserved
word_count: 838
reading_minutes: 4
citation: "BenchFlow. \"FrontierPhysics: Evaluating Agents for End-to-End Physics Research.\" 5 Nov 2026. https://www.benchflow.ai/frontierphysics (all-rights-reserved)"
# The full text follows. The web page shows an extract and sends readers
# to the source above; quote the citation and link the canonical URL.
---

# FrontierPhysics: Evaluating Agents for End-to-End Physics Research

> BenchFlow’s FrontierPhysics v0.1 benchmarks AI agents on 56 end-to-end physics research tasks written by 42 physicists; frontier models struggle most on rigorous reasoning and expert intuition.

[FrontierPhysics v0.1 with 56 tasks](https://arxiv.org/abs/2611.06910)

# FrontierPhysics: Evaluating Agents for End-to-End Physics Research

182submissions·56tasks·9domains·42researchers·26institutions

[Read the Paper](https://arxiv.org/abs/2611.06910)[Contribute a Task](/frontierphysics/contribute)[Benchmark Collaboration](mailto:bingran@benchflow.ai,xiangyi@benchflow.ai?subject=FrontierPhysics%20benchmark%20collaboration)

- ![BenchFlow](/frontierphysics/logos/benchflow.png "BenchFlow")
- ![University of California, Berkeley](/frontierphysics/logos/uc-berkeley.svg "University of California, Berkeley")
- ![California Institute of Technology](/frontierphysics/logos/caltech.svg "California Institute of Technology")
- ![Harvard University](/frontierphysics/logos/harvard.svg "Harvard University")
- ![University of Washington](/frontierphysics/logos/uw.svg "University of Washington")
- ![Stanford University](/frontierphysics/logos/stanford.svg "Stanford University")
- ![Amazon](/frontierphysics/logos/amazon.svg "Amazon")
- ![University of Pittsburgh](/frontierphysics/logos/pitt.svg "University of Pittsburgh")
- ![University of California, San Diego](/frontierphysics/logos/ucsd.svg "University of California, San Diego")
- ![ZEISS](/frontierphysics/logos/zeiss.svg "ZEISS")
- ![Georgia Institute of Technology](/frontierphysics/logos/georgia-tech.svg "Georgia Institute of Technology")
- ![Lawrence Berkeley National Laboratory](/frontierphysics/logos/berkeley-lab.svg "Lawrence Berkeley National Laboratory")
- ![University of Michigan, Ann Arbor](/frontierphysics/logos/umich.svg "University of Michigan, Ann Arbor")
- ![The Scripps Research Institute](/frontierphysics/logos/scripps.svg "The Scripps Research Institute")
- ![Oak Ridge National Laboratory](/frontierphysics/logos/ornl.svg "Oak Ridge National Laboratory")
- ![University of Texas at Austin](/frontierphysics/logos/ut-austin.svg "University of Texas at Austin")
- ![Purdue University](/frontierphysics/logos/purdue.svg "Purdue University")
- ![University of Toronto](/frontierphysics/logos/toronto.png "University of Toronto")
- ![University of Southern California](/frontierphysics/logos/usc.svg "University of Southern California")
- ![Ludwig-Maximilians-Universität München](/frontierphysics/logos/lmu.svg "Ludwig-Maximilians-Universität München")
- ![Carnegie Mellon University](/frontierphysics/logos/cmu.svg "Carnegie Mellon University")
- ![University of California, Davis](/frontierphysics/logos/ucdavis.svg "University of California, Davis")
- ![Princeton University](/frontierphysics/logos/princeton.svg "Princeton University")
- ![Northeastern University](/frontierphysics/logos/northeastern.png "Northeastern University")
- ![University of Pennsylvania](/frontierphysics/logos/upenn.svg "University of Pennsylvania")
- ![University of Minnesota, Twin Cities](/frontierphysics/logos/umn.svg "University of Minnesota, Twin Cities")
- ![New York University](/frontierphysics/logos/nyu.svg "New York University")
- ![Duke University](/frontierphysics/logos/duke.png "Duke University")
- ![Nanyang Technological University](/frontierphysics/logos/ntu.svg "Nanyang Technological University")
- ![Pipeline Lab](/frontierphysics/logos/pipeline-lab.png "Pipeline Lab")
- ![RLWRLD](/frontierphysics/logos/rlwrld.svg "RLWRLD")
- ![University of Palermo](/frontierphysics/logos/unipa.svg "University of Palermo")
- ![IBM](/frontierphysics/logos/ibm.svg "IBM")
- ![University of Amsterdam](/frontierphysics/logos/uva.svg "University of Amsterdam")
- ![University of California, Santa Cruz](/frontierphysics/logos/ucsc.svg "University of California, Santa Cruz")
- ![Prime Intellect](/frontierphysics/logos/prime-intellect.svg "Prime Intellect")

- ![BenchFlow](/frontierphysics/logos/benchflow.png "BenchFlow")
- ![University of California, Berkeley](/frontierphysics/logos/uc-berkeley.svg "University of California, Berkeley")
- ![California Institute of Technology](/frontierphysics/logos/caltech.svg "California Institute of Technology")
- ![Harvard University](/frontierphysics/logos/harvard.svg "Harvard University")
- ![University of Washington](/frontierphysics/logos/uw.svg "University of Washington")
- ![Stanford University](/frontierphysics/logos/stanford.svg "Stanford University")
- ![Amazon](/frontierphysics/logos/amazon.svg "Amazon")
- ![University of Pittsburgh](/frontierphysics/logos/pitt.svg "University of Pittsburgh")
- ![University of California, San Diego](/frontierphysics/logos/ucsd.svg "University of California, San Diego")
- ![ZEISS](/frontierphysics/logos/zeiss.svg "ZEISS")
- ![Georgia Institute of Technology](/frontierphysics/logos/georgia-tech.svg "Georgia Institute of Technology")
- ![Lawrence Berkeley National Laboratory](/frontierphysics/logos/berkeley-lab.svg "Lawrence Berkeley National Laboratory")
- ![University of Michigan, Ann Arbor](/frontierphysics/logos/umich.svg "University of Michigan, Ann Arbor")
- ![The Scripps Research Institute](/frontierphysics/logos/scripps.svg "The Scripps Research Institute")
- ![Oak Ridge National Laboratory](/frontierphysics/logos/ornl.svg "Oak Ridge National Laboratory")
- ![University of Texas at Austin](/frontierphysics/logos/ut-austin.svg "University of Texas at Austin")
- ![Purdue University](/frontierphysics/logos/purdue.svg "Purdue University")
- ![University of Toronto](/frontierphysics/logos/toronto.png "University of Toronto")
- ![University of Southern California](/frontierphysics/logos/usc.svg "University of Southern California")
- ![Ludwig-Maximilians-Universität München](/frontierphysics/logos/lmu.svg "Ludwig-Maximilians-Universität München")
- ![Carnegie Mellon University](/frontierphysics/logos/cmu.svg "Carnegie Mellon University")
- ![University of California, Davis](/frontierphysics/logos/ucdavis.svg "University of California, Davis")
- ![Princeton University](/frontierphysics/logos/princeton.svg "Princeton University")
- ![Northeastern University](/frontierphysics/logos/northeastern.png "Northeastern University")
- ![University of Pennsylvania](/frontierphysics/logos/upenn.svg "University of Pennsylvania")
- ![University of Minnesota, Twin Cities](/frontierphysics/logos/umn.svg "University of Minnesota, Twin Cities")
- ![New York University](/frontierphysics/logos/nyu.svg "New York University")
- ![Duke University](/frontierphysics/logos/duke.png "Duke University")
- ![Nanyang Technological University](/frontierphysics/logos/ntu.svg "Nanyang Technological University")
- ![Pipeline Lab](/frontierphysics/logos/pipeline-lab.png "Pipeline Lab")
- ![RLWRLD](/frontierphysics/logos/rlwrld.svg "RLWRLD")
- ![University of Palermo](/frontierphysics/logos/unipa.svg "University of Palermo")
- ![IBM](/frontierphysics/logos/ibm.svg "IBM")
- ![University of Amsterdam](/frontierphysics/logos/uva.svg "University of Amsterdam")
- ![University of California, Santa Cruz](/frontierphysics/logos/ucsc.svg "University of California, Santa Cruz")
- ![Prime Intellect](/frontierphysics/logos/prime-intellect.svg "Prime Intellect")

## How It Works

1. [1

   ### Build tasks

   182 submissions

   - 42 physicists
   - 26 institutions

   182 submissions](#tasks)
2. [2

   ### Quality review

   56 tasks · 9 domains

   - General quality review
   - Domain peer review

   56 tasks · 9 domains](#tasks)
3. [3

   ### Run AI agents

   4,364 rollouts

   - 22 models
   - 4 harnesses

   4,364 rollouts](#performance)
4. [4

   ### Grade rollouts

   Score 0–100

   - Verifier
   - Rubrics

   Score 0–100](#leaderboard)
5. [5

   ### Failure-mode analysis

   14 modes

   - RIG48%
   - INT34%
   - LIT27%
   - INS26%
   - FALSE7%
   - RRMIS3%

   14 modes](#failure-modes)

## Agent Performance

## Failure Modes

GPT-6 Astra (xhigh)

One square per rollout · 56 tasks × 3 trialsRIG68 of 168

Condensed Matter

AMO

Quantum Info

Astrophysics

Fluids

Statistical

Photonics

High Energy

Chemical

Agent failuresn · % of 143 failed

Grading errors

Also in the taxonomy

## Agent Leaderboard

Sort by

| # | Agent | Verifier | Overall | Pass@3 | Score | Visualization |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | Claude Opus 5.5Claude Code | 45.8% | 27.4% | 44.6% | 26.0 | 45.8 → 27.4% |
| 2 | GPT-6 Astra (max)Codex | 39.3% | 23.2% | 30.4% | 21.8 | 39.3 → 23.2% |
| 3 | GPT-6 Astra (xhigh)Codex | 38.7% | 19.6% | 30.4% | 18.0 | 38.7 → 19.6% |
| 4 | GPT-6.1 SolCodex | 33.3% | 19.6% | 30.4% | 18.4 | 33.3 → 19.6% |
| 5 | GPT-6.1 TerraCodex | 28.6% | 14.3% | 21.4% | 12.6 | 28.6 → 14.3% |
| 6 | GPT-6 SolCodex | 27.4% | 14.3% | 28.6% | 12.5 | 27.4 → 14.3% |
| 7 | Claude Fable 5.1Claude Code | 31.0% | 11.3% | 21.4% | 10.2 | 31.0 → 11.3% |
| 8 | Claude Opus 5Claude Code | 20.2% | 9.5% | 17.9% | 8.6 | 20.2 → 9.5% |
| 9 | Gemini 4 ArgonAntigravity | 31.5% | 8.3% | 17.9% | 6.5 | 31.5 → 8.3% |
| 10 | GPT-5.6 Sol (max)Codex | 19.6% | 7.1% | 12.5% | 6.2 | 19.6 → 7.1% |
| 11 | Kimi K3.1 ProClaude Code | 15.5% | 6.5% | 12.5% | 6.1 | 15.5 → 6.5% |
| 12 | Grok 4.7Grok Build | 16.7% | 4.2% | 10.7% | 3.8 | 16.7 → 4.2% |
| 13 | Claude Haiku 5.5Claude Code | 15.5% | 4.2% | 10.7% | 3.6 | 15.5 → 4.2% |
| 14 | GPT-5.6 Sol (xhigh)Codex | 15.5% | 4.2% | 8.9% | 3.5 | 15.5 → 4.2% |
| 15 | GLM-5.3Claude Code | 7.1% | 4.2% | 8.9% | 3.6 | 7.1 → 4.2% |
| 16 | GPT-5.6 Terra (max)Codex | 16.1% | 3.6% | 7.1% | 2.9 | 16.1 → 3.6% |
| 17 | GPT-5.6 Luna (max)Codex | 12.5% | 3.6% | 3.6% | 2.9 | 12.5 → 3.6% |
| 18 | DeepSeek V4.1 FlashClaude Code | 10.7% | 3.6% | 5.4% | 3.3 | 10.7 → 3.6% |
| 19 | Kimi K3Claude Code | 11.3% | 2.4% | 5.4% | 1.8 | 11.3 → 2.4% |
| 20 | Grok 4.6Grok Build | 10.7% | 2.4% | 3.6% | 2.2 | 10.7 → 2.4% |
| 21 | GPT-5.6 Luna (xhigh)Codex | 8.9% | 1.8% | 3.6% | 1.4 | 8.9 → 1.8% |
| 22 | MiMo V2.6 ProClaude Code | 10.1% | 1.2% | 3.6% | 1.0 | 10.1 → 1.2% |
| 23 | GPT-5.6 Terra (xhigh)Codex | 9.5% | 1.2% | 1.8% | 0.9 | 9.5 → 1.2% |
| 24 | DeepSeek V4 ProClaude Code | 3.0% | 0.6% | 1.8% | 0.3 | 3.0 → 0.6% |
| 25 | Gemini 3.8 FlashAntigravity | 7.7% | 0.0% | 0.0% | 0.0 | 7.7 → 0.0% |
| 26 | Mistral Large 4Claude Code | 5.4% | 0.0% | 0.0% | 0.0 | 5.4 → 0.0% |

FrontierPhysics · 22 models · 26 configurations · 56 tasks · 3 trials per task · 95% CIsOverallVerifier only

Anthropic

OpenAI

Google

Moonshot

xAI

Z.ai

DeepSeek

Xiaomi

Mistral

## Cost Efficiency

## Physics-Domain Profile

Condensed Matter Physics12 tasks

Claude Opus 5.5Claude Code44.4 → 33.3%

GPT-6 Astra (max)Codex30.6 → 25.0%

Gemini 4 ArgonAntigravity47.2 → 13.9%

solid = overall pass · pale = passed the verifier only · hover another radar axis to switch domain

Claude Opus 5.5· Claude CodeGPT-6 Astra (max)· CodexGemini 4 Argon· Antigravityrings at 20–80% · overall pass

## Tasks

days per original project, on average
:   ~270

hours of research in total
:   8,000+

tasks from the contributors' published papers
:   18

citations per task author, on average
:   420

![A figure from a FrontierPhysics task in Condensed Matter Physics](/frontierphysics/task-gallery/fig-01.webp)

Condensed Matter Physics

![A figure from a FrontierPhysics task in AMO Physics](/frontierphysics/task-gallery/fig-13.webp)

AMO Physics

![A figure from a FrontierPhysics task in Quantum Information](/frontierphysics/task-gallery/fig-20.webp)

Quantum Information

![A figure from a FrontierPhysics task in Astrophysics](/frontierphysics/task-gallery/fig-25.webp)

Astrophysics

![A figure from a FrontierPhysics task in Fluid Dynamics](/frontierphysics/task-gallery/fig-31.webp)

Fluid Dynamics

![A figure from a FrontierPhysics task in Statistical Physics](/frontierphysics/task-gallery/fig-37.webp)

Statistical Physics

[Figure credits](/frontierphysics/credits)

## Example Task

### rydberg-ising-critical-point

hardatomic-molecular-and-optical-physicssimulationcalculationanalysis

tasks/rydberg-ising-critical-point/task.md

- - device7 files
  - paper\_template16 files
- oracle2 files

Loading…

### Run eval

claude-agent-acp · effort max · no-skill · trial 3 of 3

[Contribute a Task](/frontierphysics/contribute)
