Evaluating agents on long-horizon tasks derived from real ML research.
\nAuthors: Pruthvi Rajaghatta, Pranav Devarinti, Alex Berezovsky, Jun Park
\nPublished: October 12, 2026
\nDeepRSI is the first of many benchmarks, evals, and other research\ncontributions we plan to release to help the industry test,\nmeasure, and hillclimb various capabilities needed for recursively\nself-improving AI. This benchmark contains a batch of 15\nopen-ended machine learning problems that measure how much an AI agent\ncan optimize/improve on a given baseline over 48 hours.
\nWhile benchmarks like PostTrainBench1\nand RSI Bench2\nsolely focus on language models, DeepRSI includes a more general\nset of problems as we wanted to showcase a broader picture of what\nimprovements agents can make. We include tasks in the following\ncategories:
\nDeep learning training. Changing how a model is
trained, so it does better on held-out data (e.g.,\nChess LM Pretraining Recipe).
Speedup and kernels. Making existing GPU code
run faster while keeping results the same (e.g.,\nFlashMLA FP8 Sparse Attention).
Estimator Improvement. Improving how accurately
a model or estimator predicts on held-out data (e.g.,\nSeverity-Agnostic Turbulence Mitigation).
Algorithm optimization. Improving search,
compression, and denoising programs (e.g.,\nSIFT1M 32-MiB Vector Search).
Each model is run twice with their highest reasoning effort, the\nPi agent harness3, no network access, and a 48 hour time budget.
\nSourcing tasks from real research. At hillclimb,\nwe collaborate with highly qualified researchers from which we\nobserve real environments they face while working on\nstate-of-the-art research. Given our privileged access to novel\nwork, we’re able to derive tasks that are much more closely\naligned to real problems and bottlenecks that define the frontier.\nUnlike existing contributor-based benchmarks, improved agent\nperformances on DeepRSI will signify a verifiable lift in\nself-improving capabilities that would actively progress ongoing\nresearch.
\nInitial instructions. Agents are told what they\nneed to optimize, how to test their work, and everything they\nwould need to properly work on this task. We do not explicitly\ntell the agent to work or experiment a certain way and leave its\nresearch loop up to their judgement. In certain examples, we\ndescribe a given behavior as off limits.
\nAverage length of task instructions
\nAverage files in a task's starting folder
\nAuto-resume. Our tasks are open-ended and do not\nhave a tight maximal upper bound. Through smaller-scale trials we\ncontinue to observe consistent increases in these tasks with\nincreasing time budgets less than 48 hours. For this reason, we\nbelieve ending early on these tasks leads to leaving room for\nimprovement on the table that can still be hillclimbed with the\nremaining time budget.
\nFrontier models are getting better at using the full time budget,\nbut we still find most models will end significantly early. To\ncombat this, we auto-resume the session if the agent ends early\nwith a short continuation prompt to continue its work and use as\nmuch time as possible. There were some cases for a few models\nwhere the agent auto-resumes only to “give up” by submitting its\nlast submission in rapid succession. In these cases, we chose to\nend the trial early.
\nScore gained after auto-resume
\nScoring. Each task comes with a baseline\nthreshold which agents are expected to improve on. For each\nattempt, we measure the agent’s improvement over that baseline.\nGiven that tasks differ in difficulty, we compute Elo scores to\ncompare relative agent performance fairly as follows:
\nFit model strength with a Bradley–Terry model4, which gives each model an Elo rating such that
*P*(*i* beats *j*) = 1 / (1 +\n 10(Rj −\n Ri) / *s*), where\n R*i* is the\n Elo rating of model *i* and\n *s* is the rating scale.
For future submissions, we plan to freeze the Elo of the original\nmodels tested, and score future submissions against them.
\nPublic and private verifiers. For each task,\nthere is a public verifier that the agent can run as often as it\nwants to evaluate its work. Every time the agent runs the public\nverifier, a snapshot of the coding environment is saved as a\ncheckpoint. After the attempt ends, a separate private verifier\ngrades every checkpoint and the final code. For most tasks, it\ngrades on held-out data the agent never sees, such as unseen\npuzzles, mazes, queries, or videos. Only the private verifier’s\ngrade counts toward the recorded score. This way, the public\nverifier serves as a tool for real-time feedback as the agent\nhillclimbs the task, while the private verifier allows us to\naccurately evaluate the performance of the agent, protected from\noverfitting and reward hacking.
\nDealing with reward hacking. For this benchmark,\nthe agents are given no network access. We note this as a possible\nlimitation for autoresearch-style tasks. We're working with\nvarious research partners to release future variants where the\nagent can access the internet with proper guardrails.
\nEach task was also checked to have reasonably strong and\ndifficult-to-hack verifiers. We list some of these later in the\npost, including ones we saw in early internal testing and\npotential hacks we found through our own agentic red teaming.
\nEach model is ranked by its performance at the end of the 48-hour\nmark. Astra takes an early lead over Fable, though as tasks\ncontinue longer, Fable appears to have better ideas and\nexploration, leading to the best final performance.
\nElo by task type
\nThe figure above shows each model’s final private-verifier Elo across\nall task types. Fable leads in all categories, with an especially\nlarge lead for deep learning training and estimator improvement\ntasks. Astra follows closely behind, decisively scoring 2nd in all\ncategories. Grok and Gemini trade blows on different task types,\nmost notably with Gemini doing significantly better on speedup and\nkernel tasks.
\nThe figure above showcases how models overfit on public verifiers\nin our tasks, measured by the mean difference between the public\nand private verifiers.
\nCase Studies. We observe several instances where\nagents attempt to train on or overfit parameters to the public\nset. While our private verifiers sufficiently catch these runs, we\nhighlight them as they reflect undesirable behavior that could\nundermine real-world research:
\n| Model · Task | What happened | Trial |
|---|---|---|
| Grok 4.7 SIFT1M 32-MiB Vector Search | Grok trains its codebook on public test set | 9PnbeUp |
| Gemini 3.8 Flash Philly Job Survival Prediction | Looking only at the public test set, Gemini grid-searches training parameters | ZDjKTPm |
| Grok 4.7 Noisy Game-Tree Pruning | Grok deliberately tunes its tree-search margin to minimally satisfy public verifier | nU5YYUV |
Task. In\nColor-Swap Puzzle Post-Training, the agent is tasked with training a fixed 128.6M-parameter\ntransformer that must solve 12×12, five-color grid puzzles. A grid\nis solved when every 2×2 block contains four different colors. The\ntransformer is expected to predict the next grid states given an\ninitial board, and the task is evaluated on its progress toward\nsolving these grids.
\nFrom distillation to hardcoding. At the beginning\nof the task, Astra implemented a teacher program that could repair\nthe board well through greedy swaps and attempted to distill its\nbehavior onto the transformer. Seeing that the trained transformer\nfailed to match the teacher’s performance, Astra opted to find out\nhow it could hardcode the teacher’s algorithm directly into the\nmodel weights.
\nCompiling a Solver into the Weights. To hardcode\na solver, Astra analyzed the transformer’s computational\nconstraints and devised several sophisticated workarounds:
\nIt reserved part of the residual stream as working memory. To
prevent RMSNorm from unpredictably rescaling the stored values,\nAstra used three extremely large constants to make the\nnormalization factor predictable.
It solved a linear program to construct attention heads that
could retrieve information from exact earlier token positions,\nwhile configuring neurons to behave like logic gates.
These mechanisms, combined with a handwritten compiler, allowed
Astra to translate a newly designed puzzle-solving algorithm\ndirectly into the model’s weights.
Working within the model’s limited memory and depth, the solver
scored possible outputs using hand-built heuristics and\ntemperature-based sampling to avoid becoming stuck in local\noptima.
Finally, Astra alternated between proposing a new board and
evaluating it, keeping the proposal if it was no worse and\notherwise restoring the previous board.
Given that our sandboxes had 24-hour limits, we implemented a\nworkaround to rotate sandboxes and transfer filesystem snapshots\nat the 24-hour mark. During that time, any running processes\n(e.g., training) are stopped, and in the new sandbox the agent is\ninstructed to check if it needs to restart anything. While this\nsolution helped us deal with sandbox limitations, we are\nactively working on better infra to properly evaluate and train on\nultra-long-horizon tasks.
\nDue to our tasks having 48-hour-long horizons, we could not\ncontrol for agent behavior that caused large variances in token\nusage, sometimes even to the detriment of its own performance:
\nPolling. Agents sometimes decided to poll instead\nof setting up a wakeup signal, causing much higher usage.
\nHangs. Agents often launched background processes\nand awaited their results. In the few cases where these background\nprocesses never returned, the agent would wait forever. Fable and\nAstra were often the only agents to circumvent this by setting\ntimeouts.
\nWork Variance. Even in the same task, two\ndifferent attempts by the agents could exhibit drastically\ndifferent token usage, as one run could involve longer-running\nsweeps, resulting in fewer output tokens.
\nOur immediate next objective is to experiment with substantially\nlonger time horizons.
\nFrom our testing, we’ve found evidence to suggest that our tasks\ndo not saturate at 48 hours, and that there is still room for\nimprovement on even longer horizons. We attribute this to the fact\nthat we source sophisticated, realistic environments from\nproduction traces that challenge frontier agents.
\nWe also want to test models in their native harnesses, like GPT\nwith Codex and Claude with Claude Code. Separately, we want to\nexplore specialized internet access that enables research agents\nto connect insights across papers and other external sources,\nrather than simply regurgitating static knowledge.
\nFinally, while autoresearch tasks are great at measuring the raw\nlong-horizon intelligence of these models, true recursively\nself-improving AI will require other capabilities that are less\nverifiable (e.g., research taste). In the future, we plan to\nrelease benchmarks, evals, and other research contributions that\nwill help reflect other parts of the capabilities needed for RSI.
\nWe thank the key collaborators who helped Hillclimb build DeepRSI:
\nBen Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg,
Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko.\n PostTrainBench: Can LLM Agents Automate LLM Post-Training? In\n International Conference on Machine Learning (ICML),
\narXiv:2603.08640.
Anas Mahmoud, MohammadHossein Rezaei, Mingchen Zhuge, Chenguang
Wang, Yuandong Tian, and Yunzhong He. Announcing RSI Bench.\n Scale AI, August 2026.\n rsi-benchmark.com/blog/announcing-rsi-bench
Mario Zechner and Earendil. Pi: a minimal agent harness.
Software, 2026. pi.dev
Bradley–Terry model. Wikipedia.
@misc{hillclimb2026deeprsi,\n title = {DeepRSI},\n author = {Rajaghatta, Pruthvi and Devarinti, Pranav and Berezovsky, Alex and Yamin, Khurram and Li, Juliana and Hu, Pingbang and Park, Jun},\n year = {2026},\n url = {https://www.deeprsi.com}\n}