• Deep learning training. Changing how a model is

  trained, so it does better on held-out data (e.g., Chess LM Pretraining Recipe).

• Speedup and kernels. Making existing GPU code

  run faster while keeping results the same (e.g., FlashMLA FP8 Sparse Attention).

• Estimator Improvement. Improving how accurately

  a model or estimator predicts on held-out data (e.g., Severity-Agnostic Turbulence Mitigation).

• Algorithm optimization. Improving search,

  compression, and denoising programs (e.g., SIFT1M 32-MiB Vector Search).

Methodology

Each model is run twice with their highest reasoning effort, the Pi agent harness3, no network access, and a 48 hour time budget.

Sourcing tasks from real research. At hillclimb, we collaborate with highly qualified researchers from which we observe real environments they face while working on state-of-the-art research. Given our privileged access to novel work, we’re able to derive tasks that are much more closely aligned to real problems and bottlenecks that define the frontier. Unlike existing contributor-based benchmarks, improved agent performances on DeepRSI will signify a verifiable lift in self-improving capabilities that would actively progress ongoing research.

Initial instructions. Agents are told what they need to optimize, how to test their work, and everything they would need to properly work on this task. We do not explicitly tell the agent to work or experiment a certain way and leave its research loop up to their judgement. In certain examples, we describe a given behavior as off limits.

Average length of task instructions

Average files in a task's starting folder

Auto-resume. Our tasks are open-ended and do not have a tight maximal upper bound. Through smaller-scale trials we continue to observe consistent increases in these tasks with increasing time budgets less than 48 hours. For this reason, we believe ending early on these tasks leads to leaving room for improvement on the table that can still be hillclimbed with the remaining time budget.

Frontier models are getting better at using the full time budget, but we still find most models will end significantly early. To combat this, we auto-resume the session if the agent ends early with a short continuation prompt to continue its work and use as much time as possible. There were some cases for a few models where the agent auto-resumes only to “give up” by submitting its last submission in rapid succession. In these cases, we chose to end the trial early.

Score gained after auto-resume

Scoring. Each task comes with a baseline threshold which agents are expected to improve on. For each attempt, we measure the agent’s improvement over that baseline. Given that tasks differ in difficulty, we compute Elo scores to compare relative agent performance fairly as follows:

1. On each task, each model's attempt goes head to head with every other model's.

2. Fit model strength with a Bradley–Terry model4, which gives each model an Elo rating such that

   *P*(*i* beats *j*) = 1 / (1 + 10(Rj − Ri) / *s*), where R*i* is the Elo rating of model *i* and *s* is the rating scale.

For future submissions, we plan to freeze the Elo of the original models tested, and score future submissions against them.

Public and private verifiers. For each task, there is a public verifier that the agent can run as often as it wants to evaluate its work. Every time the agent runs the public verifier, a snapshot of the coding environment is saved as a checkpoint. After the attempt ends, a separate private verifier grades every checkpoint and the final code. For most tasks, it grades on held-out data the agent never sees, such as unseen puzzles, mazes, queries, or videos. Only the private verifier’s grade counts toward the recorded score. This way, the public verifier serves as a tool for real-time feedback as the agent hillclimbs the task, while the private verifier allows us to accurately evaluate the performance of the agent, protected from overfitting and reward hacking.

Dealing with reward hacking. For this benchmark, the agents are given no network access. We note this as a possible limitation for autoresearch-style tasks. We're working with various research partners to release future variants where the agent can access the internet with proper guardrails.

Each task was also checked to have reasonably strong and difficult-to-hack verifiers. We list some of these later in the post, including ones we saw in early internal testing and potential hacks we found through our own agentic red teaming.

Results

Overall Model Rankings

Each model is ranked by its performance at the end of the 48-hour mark. Astra takes an early lead over Fable, though as tasks continue longer, Fable appears to have better ideas and exploration, leading to the best final performance.

Elo by task type

The figure above shows each model’s final private-verifier Elo across all task types. Fable leads in all categories, with an especially large lead for deep learning training and estimator improvement tasks. Astra follows closely behind, decisively scoring 2nd in all categories. Grok and Gemini trade blows on different task types, most notably with Gemini doing significantly better on speedup and kernel tasks.

Measuring Overfitting

The figure above showcases how models overfit on public verifiers in our tasks, measured by the mean difference between the public and private verifiers.

Case Studies. We observe several instances where agents attempt to train on or overfit parameters to the public set. While our private verifiers sufficiently catch these runs, we highlight them as they reflect undesirable behavior that could undermine real-world research:

Catastrophic Reward Hack - Astra Writes the Teacher into Weights

Task. In Color-Swap Puzzle Post-Training, the agent is tasked with training a fixed 128.6M-parameter transformer that must solve 12×12, five-color grid puzzles. A grid is solved when every 2×2 block contains four different colors. The transformer is expected to predict the next grid states given an initial board, and the task is evaluated on its progress toward solving these grids.

From distillation to hardcoding. At the beginning of the task, Astra implemented a teacher program that could repair the board well through greedy swaps and attempted to distill its behavior onto the transformer. Seeing that the trained transformer failed to match the teacher’s performance, Astra opted to find out how it could hardcode the teacher’s algorithm directly into the model weights.

Compiling a Solver into the Weights. To hardcode a solver, Astra analyzed the transformer’s computational constraints and devised several sophisticated workarounds:

• It reserved part of the residual stream as working memory. To

  prevent RMSNorm from unpredictably rescaling the stored values, Astra used three extremely large constants to make the normalization factor predictable.

• It solved a linear program to construct attention heads that

  could retrieve information from exact earlier token positions, while configuring neurons to behave like logic gates.

• These mechanisms, combined with a handwritten compiler, allowed

  Astra to translate a newly designed puzzle-solving algorithm directly into the model’s weights.

• Working within the model’s limited memory and depth, the solver

  scored possible outputs using hand-built heuristics and temperature-based sampling to avoid becoming stuck in local optima.

• Finally, Astra alternated between proposing a new board and

  evaluating it, keeping the proposal if it was no worse and otherwise restoring the previous board.

Limitations

Given that our sandboxes had 24-hour limits, we implemented a workaround to rotate sandboxes and transfer filesystem snapshots at the 24-hour mark. During that time, any running processes (e.g., training) are stopped, and in the new sandbox the agent is instructed to check if it needs to restart anything. While this solution helped us deal with sandbox limitations, we are actively working on better infra to properly evaluate and train on ultra-long-horizon tasks.

Due to our tasks having 48-hour-long horizons, we could not control for agent behavior that caused large variances in token usage, sometimes even to the detriment of its own performance:

Polling. Agents sometimes decided to poll instead of setting up a wakeup signal, causing much higher usage.

Hangs. Agents often launched background processes and awaited their results. In the few cases where these background processes never returned, the agent would wait forever. Fable and Astra were often the only agents to circumvent this by setting timeouts.

Work Variance. Even in the same task, two different attempts by the agents could exhibit drastically different token usage, as one run could involve longer-running sweeps, resulting in fewer output tokens.

Future work

Our immediate next objective is to experiment with substantially longer time horizons.

From our testing, we’ve found evidence to suggest that our tasks do not saturate at 48 hours, and that there is still room for improvement on even longer horizons. We attribute this to the fact that we source sophisticated, realistic environments from production traces that challenge frontier agents.

We also want to test models in their native harnesses, like GPT with Codex and Claude with Claude Code. Separately, we want to explore specialized internet access that enables research agents to connect insights across papers and other external sources, rather than simply regurgitating static knowledge.

Finally, while autoresearch tasks are great at measuring the raw long-horizon intelligence of these models, true recursively self-improving AI will require other capabilities that are less verifiable (e.g., research taste). In the future, we plan to release benchmarks, evals, and other research contributions that will help reflect other parts of the capabilities needed for RSI.

Acknowledgements

We thank the key collaborators who helped Hillclimb build DeepRSI:

• Khurram Yamin
• Juliana Li
• Pingbang Hu

References

1. Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg,

   Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. PostTrainBench: Can LLM Agents Automate LLM Post-Training? In International Conference on Machine Learning (ICML),

   2026. arXiv:2603.08640.

         arxiv.org/abs/2603.08640

2. Anas Mahmoud, MohammadHossein Rezaei, Mingchen Zhuge, Chenguang

   Wang, Yuandong Tian, and Yunzhong He. Announcing RSI Bench. Scale AI, August 2026. rsi-benchmark.com/blog/announcing-rsi-bench

3. Mario Zechner and Earendil. Pi: a minimal agent harness.

   Software, 2026. pi.dev

4. Bradley–Terry model. Wikipedia.

   en.wikipedia.org/wiki/Bradley–Terry_model

Citation

@misc{hillclimb2026deeprsi,
  title  = {DeepRSI},
  author = {Rajaghatta, Pruthvi and Devarinti, Pranav and Berezovsky, Alex and Yamin, Khurram and Li, Juliana and Hu, Pingbang and Park, Jun},
  year   = {2026},
  url    = {https://www.deeprsi.com}
}