{"article":{"slug":"two-hills-weights-and-harnesses","title":"Two Hills: Weights and Harnesses","subtitle":null,"summary":"Part 1 of an LLM agent evaluation series: when a model system is not good enough, you face two hills—improving model weights versus improving the harness around the model—and how to choose where to invest.","content_type":"blog_post","language":"en","canonical_url":"https://rlafuente.com/posts/2026-10-11-agent-evals-1-two-hills.html","author":{"name":"Rodney L.","url":"https://rlafuente.com/","person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Rodney L.","url":"https://rlafuente.com/","listing_slug":null,"listing":null},"topics":[{"name":"AI","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"AI Agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"Benchmarks","slug":"benchmarks","url":"https://listedarticles.com/topics/benchmarks"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":1328,"reading_minutes":6,"published_at":"2026-10-11T12:00:00.000Z","added_at":"2026-10-12T05:11:21.288Z","updated_at":"2026-10-12T05:11:21.288Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/two-hills-weights-and-harnesses","markdown_url":"https://listedarticles.com/articles/two-hills-weights-and-harnesses.md","example":false,"citation":"Rodney L., Rodney L.. \"Two Hills: Weights and Harnesses.\" 11 Oct 2026. https://rlafuente.com/posts/2026-10-11-agent-evals-1-two-hills.html (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://rlafuente.com/posts/2026-10-11-agent-evals-1-two-hills.html"},"body_markdown":"# Two Hills: Weights and Harnesses\n\nPublished 2026-10-11\n\nEvaluating LLM Agents, part 1. [Series page and full outline.](<https://rlafuente.com/agent-evals>)\n\nDefers: how to build the eval itself to Parts 2 through 5, judge calibration to Part 7, and statistics for small eval sets to Part 8 (\n\n[outline](<https://rlafuente.com/agent-evals#outline>)).\n\nWhen a language model system isn't good enough, there are two kinds of change that could fix it. You can change the weights, or you can change the agent harness, the code around the model that assembles its prompt, exposes its tools, manages context and memory, and runs the loop that decides when to call a tool, retry, or stop. Claude Code and SWE-agent are harnesses in this sense, and Anthropic's engineering team is explicit that when they evaluate an agent, they're evaluating \"the harness and the model working together\" ([Anthropic, 2026](<https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents>)). An evaluation harness, the infrastructure that runs the evals, is a separate thing, and so is the sandbox the agent acts in, which Part 5 treats as part of the environment.\n\nI think of these as two different hills, and the uncomfortable part is that we only really know how to climb one of them.\n\nFigure 1. On the weights hill, the measurement and the optimizer are the same object. On the harness hill, a person makes a discrete change and an eval tells them whether it helped.\n\n## The weights hill has a gradient\n\nEvery method I know of for improving weights has the same basic shape. Some scalar signal comes from somewhere, it gets turned into a gradient, and a step gets taken in that direction, over and over. The only thing that really varies from one method to the next is where that signal comes from:\n\n  * Pretraining and supervised fine-tuning use a loss against data.\n  * RLHF gets its signal from a reward model trained on human preferences ([Christiano et al., 2017](<https://arxiv.org/abs/1706.03741>); [Ouyang et al., 2022](<https://arxiv.org/abs/2203.02155>)); DPO is the same idea with the reward model folded into the loss, so there's nothing separate to train ([Rafailov et al., 2023](<https://arxiv.org/abs/2305.18290>)).\n  * RLVR throws the reward model away and uses a program instead: a unit test, a symbolic match against a known answer, whatever can be checked mechanically ([Lambert et al., 2024](<https://arxiv.org/abs/2411.15124>); [DeepSeek-AI, 2025](<https://arxiv.org/abs/2501.12948>)).\n\n\n\nIn all of these the reward is also the eval, and the optimizer takes millions of tiny steps against it with nobody watching. It's a mature discipline. There are loss curves to look at, learning rates to tune, and a long history of optimization theory to lean on when something goes wrong.\n\nNone of which makes it safe. Push hard enough on any proxy and it stops meaning what it meant when you started ([Gao et al., 2022](<https://arxiv.org/abs/2210.10760>)), and the signal can even detach from capability completely: [Shao et al. (2025)](<https://arxiv.org/abs/2506.10947>) got a 21.4-point improvement on MATH-500 out of Qwen2.5-Math-7B using rewards assigned at random, not far off what the real rewards gave. So even on the weights hill, the signal you optimize and the signal you trust had better be two different things.\n\n## The harness hill has no gradient\n\nA harness, by contrast, is a handful of discrete, structured choices: a paragraph in a prompt, a tool, a rule about when to stop. There's no way to differentiate through \"add a retry.\" Every step up this hill is a person making a change and then measuring what happened, and there's no loss curve to watch while they do it. The eval is the only thing telling them which way is up.\n\nPeople have tried to manufacture a gradient, and some of the attempts are clever. ProTeGi forms natural-language \"gradients\" that criticize a prompt and then edits against them ([Pryzant et al., 2023](<https://arxiv.org/abs/2305.03495>)). OPRO uses the model itself as the optimizer ([Yang et al., 2023](<https://arxiv.org/abs/2309.03409>)). DSPy compiles whole pipelines against a metric ([Khattab et al., 2023](<https://arxiv.org/abs/2310.03714>)), and TextGrad propagates text feedback backwards through a program ([Yuksekgonul et al., 2024](<https://arxiv.org/abs/2406.07496>)). Reflexion improves an agent \"not by updating weights, but instead through linguistic feedback\" ([Shinn et al., 2023](<https://arxiv.org/abs/2303.11366>)). What all of them have in common is that they need a metric to optimize. None of them gets rid of the eval; they just turn it into the objective function.\n\nWhich is why I'm writing this series. On the harness hill, the eval has to do the job the gradient does on the weights hill. If it's noisy, you take random steps. If it measures the wrong thing, you walk up the wrong hill, and nothing in the process will flag it, because there's no held-out loss to diverge.\n\n## Why this hill is new\n\nAgent harnesses as we now know them barely existed before ReAct put a reasoning-and-acting loop around a model in late 2022 ([Yao et al., 2022](<https://arxiv.org/abs/2210.03629>)). The paper that convinced me the harness is its own axis was SWE-agent: with the same class of models everyone else had, a redesigned agent-computer interface got to 12.5% pass@1 on SWE-bench, well above what non-interactive use of those models had managed ([Yang et al., 2024](<https://arxiv.org/abs/2405.15793>)). Then [Kapoor et al. (2024)](<https://arxiv.org/abs/2407.01502>) argued that the field had been drawing the wrong conclusions about where its accuracy gains came from, because nobody was measuring cost alongside accuracy. More recently the Holistic Agent Leaderboard ran 21,730 rollouts across models, scaffolds, and benchmarks and found, among other things, that higher reasoning effort made accuracy worse in the majority of runs ([Kapoor et al., 2025](<https://arxiv.org/abs/2510.11977>)).\n\nSo the practice of measuring harness changes is maybe three years old, and most of the tooling is younger than that. The Model Context Protocol, which is what standardized how tools get exposed to a model, only came out in late 2024 ([Anthropic, 2024](<https://modelcontextprotocol.io/>)).\n\n## Who is on which hill\n\nMost teams that build with language models never touch weights at all. They're on the harness hill exclusively, which means the eval is the whole of their optimization machinery, whether or not they think of it that way. Labs climb both hills, and lately the two have been converging: the executable environments used to train coding agents with RL are the same environments people use to evaluate harnesses ([Pan et al., 2024](<https://arxiv.org/abs/2412.21139>); [Yang et al., 2025](<https://arxiv.org/abs/2504.21798>)). Part 5 comes back to that.\n\nThe rest of the series is about how to build the measurement.\n\n  * [Measure a workflow, not a system.](<https://rlafuente.com/posts/2026-10-11-agent-evals-2-measure-a-workflow>)\n  * [What changes when the model acts.](<https://rlafuente.com/posts/2026-10-11-agent-evals-3-agents-are-not-llms>)\n  * [Verifiers first, rubrics second.](<https://rlafuente.com/posts/2026-10-11-agent-evals-4-verifiers-first-rubrics-second>)\n  * [The environment is the eval.](<https://rlafuente.com/posts/2026-10-11-agent-evals-5-the-environment-is-the-eval>)\n\n\n\nSeven more parts are outlined on the [series page](<https://rlafuente.com/agent-evals#outline>).\n\n## References\n\nAnthropic (2026). [Demystifying evals for AI agents](<https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents>). Anthropic Engineering.  \nChristiano, P. F. et al. (2017). [Deep Reinforcement Learning from Human Preferences](<https://arxiv.org/abs/1706.03741>). NeurIPS.  \nOuyang, L. et al. (2022). [Training language models to follow instructions with human feedback](<https://arxiv.org/abs/2203.02155>). NeurIPS.  \nRafailov, R. et al. (2023). [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](<https://arxiv.org/abs/2305.18290>). NeurIPS.  \nLambert, N. et al. (2024). [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](<https://arxiv.org/abs/2411.15124>).  \nDeepSeek-AI (2025). [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](<https://arxiv.org/abs/2501.12948>).  \nGao, L., Schulman, J., and Hilton, J. (2022). [Scaling Laws for Reward Model Overoptimization](<https://arxiv.org/abs/2210.10760>). ICML 2023.  \nShao, R. et al. (2025). [Spurious Rewards: Rethinking Training Signals in RLVR](<https://arxiv.org/abs/2506.10947>).  \nPryzant, R. et al. (2023). [Automatic Prompt Optimization with \"Gradient Descent\" and Beam Search](<https://arxiv.org/abs/2305.03495>). EMNLP.  \nYang, C. et al. (2023). [Large Language Models as Optimizers](<https://arxiv.org/abs/2309.03409>). ICLR 2024.  \nKhattab, O. et al. (2023). [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](<https://arxiv.org/abs/2310.03714>). ICLR 2024.  \nYuksekgonul, M. et al. (2024). [TextGrad: Automatic \"Differentiation\" via Text](<https://arxiv.org/abs/2406.07496>).  \nShinn, N. et al. (2023). [Reflexion: Language Agents with Verbal Reinforcement Learning](<https://arxiv.org/abs/2303.11366>). NeurIPS.  \nYao, S. et al. (2022). [ReAct: Synergizing Reasoning and Acting in Language Models](<https://arxiv.org/abs/2210.03629>). ICLR 2023.  \nYang, J. et al. (2024). [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](<https://arxiv.org/abs/2405.15793>). NeurIPS.  \nKapoor, S. et al. (2024). [AI Agents That Matter](<https://arxiv.org/abs/2407.01502>).  \nKapoor, S. et al. (2025). [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](<https://arxiv.org/abs/2510.11977>).  \nAnthropic (2024). [Model Context Protocol](<https://modelcontextprotocol.io/>). Specification and documentation.  \nPan, J. et al. (2024). [Training Software Engineering Agents and Verifiers with SWE-Gym](<https://arxiv.org/abs/2412.21139>).  \nYang, J. et al. (2025). [SWE-smith: Scaling Data for Software Engineering Agents](<https://arxiv.org/abs/2504.21798>).","body_html":"<h1 id=\"two-hills-weights-and-harnesses\">Two Hills: Weights and Harnesses</h1>\n<p>Published 2026-10-11</p>\n<p>Evaluating LLM Agents, part 1. <a href=\"https://rlafuente.com/agent-evals\" rel=\"nofollow ugc noopener\">Series page and full outline.</a></p>\n<p>Defers: how to build the eval itself to Parts 2 through 5, judge calibration to Part 7, and statistics for small eval sets to Part 8 (</p>\n<p><a href=\"https://rlafuente.com/agent-evals#outline\" rel=\"nofollow ugc noopener\">outline</a>).</p>\n<p>When a language model system isn&#39;t good enough, there are two kinds of change that could fix it. You can change the weights, or you can change the agent harness, the code around the model that assembles its prompt, exposes its tools, manages context and memory, and runs the loop that decides when to call a tool, retry, or stop. Claude Code and SWE-agent are harnesses in this sense, and Anthropic&#39;s engineering team is explicit that when they evaluate an agent, they&#39;re evaluating &quot;the harness and the model working together&quot; (<a href=\"https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\" rel=\"nofollow ugc noopener\">Anthropic, 2026</a>). An evaluation harness, the infrastructure that runs the evals, is a separate thing, and so is the sandbox the agent acts in, which Part 5 treats as part of the environment.</p>\n<p>I think of these as two different hills, and the uncomfortable part is that we only really know how to climb one of them.</p>\n<p>Figure 1. On the weights hill, the measurement and the optimizer are the same object. On the harness hill, a person makes a discrete change and an eval tells them whether it helped.</p>\n<h2 id=\"the-weights-hill-has-a-gradient\">The weights hill has a gradient</h2>\n<p>Every method I know of for improving weights has the same basic shape. Some scalar signal comes from somewhere, it gets turned into a gradient, and a step gets taken in that direction, over and over. The only thing that really varies from one method to the next is where that signal comes from:</p>\n<ul><li>Pretraining and supervised fine-tuning use a loss against data.</li><li>RLHF gets its signal from a reward model trained on human preferences (<a href=\"https://arxiv.org/abs/1706.03741\" rel=\"nofollow ugc noopener\">Christiano et al., 2017</a>; <a href=\"https://arxiv.org/abs/2203.02155\" rel=\"nofollow ugc noopener\">Ouyang et al., 2022</a>); DPO is the same idea with the reward model folded into the loss, so there&#39;s nothing separate to train (<a href=\"https://arxiv.org/abs/2305.18290\" rel=\"nofollow ugc noopener\">Rafailov et al., 2023</a>).</li><li>RLVR throws the reward model away and uses a program instead: a unit test, a symbolic match against a known answer, whatever can be checked mechanically (<a href=\"https://arxiv.org/abs/2411.15124\" rel=\"nofollow ugc noopener\">Lambert et al., 2024</a>; <a href=\"https://arxiv.org/abs/2501.12948\" rel=\"nofollow ugc noopener\">DeepSeek-AI, 2025</a>).</li></ul>\n<p>In all of these the reward is also the eval, and the optimizer takes millions of tiny steps against it with nobody watching. It&#39;s a mature discipline. There are loss curves to look at, learning rates to tune, and a long history of optimization theory to lean on when something goes wrong.</p>\n<p>None of which makes it safe. Push hard enough on any proxy and it stops meaning what it meant when you started (<a href=\"https://arxiv.org/abs/2210.10760\" rel=\"nofollow ugc noopener\">Gao et al., 2022</a>), and the signal can even detach from capability completely: <a href=\"https://arxiv.org/abs/2506.10947\" rel=\"nofollow ugc noopener\">Shao et al. (2025)</a> got a 21.4-point improvement on MATH-500 out of Qwen2.5-Math-7B using rewards assigned at random, not far off what the real rewards gave. So even on the weights hill, the signal you optimize and the signal you trust had better be two different things.</p>\n<h2 id=\"the-harness-hill-has-no-gradient\">The harness hill has no gradient</h2>\n<p>A harness, by contrast, is a handful of discrete, structured choices: a paragraph in a prompt, a tool, a rule about when to stop. There&#39;s no way to differentiate through &quot;add a retry.&quot; Every step up this hill is a person making a change and then measuring what happened, and there&#39;s no loss curve to watch while they do it. The eval is the only thing telling them which way is up.</p>\n<p>People have tried to manufacture a gradient, and some of the attempts are clever. ProTeGi forms natural-language &quot;gradients&quot; that criticize a prompt and then edits against them (<a href=\"https://arxiv.org/abs/2305.03495\" rel=\"nofollow ugc noopener\">Pryzant et al., 2023</a>). OPRO uses the model itself as the optimizer (<a href=\"https://arxiv.org/abs/2309.03409\" rel=\"nofollow ugc noopener\">Yang et al., 2023</a>). DSPy compiles whole pipelines against a metric (<a href=\"https://arxiv.org/abs/2310.03714\" rel=\"nofollow ugc noopener\">Khattab et al., 2023</a>), and TextGrad propagates text feedback backwards through a program (<a href=\"https://arxiv.org/abs/2406.07496\" rel=\"nofollow ugc noopener\">Yuksekgonul et al., 2024</a>). Reflexion improves an agent &quot;not by updating weights, but instead through linguistic feedback&quot; (<a href=\"https://arxiv.org/abs/2303.11366\" rel=\"nofollow ugc noopener\">Shinn et al., 2023</a>). What all of them have in common is that they need a metric to optimize. None of them gets rid of the eval; they just turn it into the objective function.</p>\n<p>Which is why I&#39;m writing this series. On the harness hill, the eval has to do the job the gradient does on the weights hill. If it&#39;s noisy, you take random steps. If it measures the wrong thing, you walk up the wrong hill, and nothing in the process will flag it, because there&#39;s no held-out loss to diverge.</p>\n<h2 id=\"why-this-hill-is-new\">Why this hill is new</h2>\n<p>Agent harnesses as we now know them barely existed before ReAct put a reasoning-and-acting loop around a model in late 2022 (<a href=\"https://arxiv.org/abs/2210.03629\" rel=\"nofollow ugc noopener\">Yao et al., 2022</a>). The paper that convinced me the harness is its own axis was SWE-agent: with the same class of models everyone else had, a redesigned agent-computer interface got to 12.5% pass@1 on SWE-bench, well above what non-interactive use of those models had managed (<a href=\"https://arxiv.org/abs/2405.15793\" rel=\"nofollow ugc noopener\">Yang et al., 2024</a>). Then <a href=\"https://arxiv.org/abs/2407.01502\" rel=\"nofollow ugc noopener\">Kapoor et al. (2024)</a> argued that the field had been drawing the wrong conclusions about where its accuracy gains came from, because nobody was measuring cost alongside accuracy. More recently the Holistic Agent Leaderboard ran 21,730 rollouts across models, scaffolds, and benchmarks and found, among other things, that higher reasoning effort made accuracy worse in the majority of runs (<a href=\"https://arxiv.org/abs/2510.11977\" rel=\"nofollow ugc noopener\">Kapoor et al., 2025</a>).</p>\n<p>So the practice of measuring harness changes is maybe three years old, and most of the tooling is younger than that. The Model Context Protocol, which is what standardized how tools get exposed to a model, only came out in late 2024 (<a href=\"https://modelcontextprotocol.io/\" rel=\"nofollow ugc noopener\">Anthropic, 2024</a>).</p>\n<h2 id=\"who-is-on-which-hill\">Who is on which hill</h2>\n<p>Most teams that build with language models never touch weights at all. They&#39;re on the harness hill exclusively, which means the eval is the whole of their optimization machinery, whether or not they think of it that way. Labs climb both hills, and lately the two have been converging: the executable environments used to train coding agents with RL are the same environments people use to evaluate harnesses (<a href=\"https://arxiv.org/abs/2412.21139\" rel=\"nofollow ugc noopener\">Pan et al., 2024</a>; <a href=\"https://arxiv.org/abs/2504.21798\" rel=\"nofollow ugc noopener\">Yang et al., 2025</a>). Part 5 comes back to that.</p>\n<p>The rest of the series is about how to build the measurement.</p>\n<ul><li><a href=\"https://rlafuente.com/posts/2026-10-11-agent-evals-2-measure-a-workflow\" rel=\"nofollow ugc noopener\">Measure a workflow, not a system.</a></li><li><a href=\"https://rlafuente.com/posts/2026-10-11-agent-evals-3-agents-are-not-llms\" rel=\"nofollow ugc noopener\">What changes when the model acts.</a></li><li><a href=\"https://rlafuente.com/posts/2026-10-11-agent-evals-4-verifiers-first-rubrics-second\" rel=\"nofollow ugc noopener\">Verifiers first, rubrics second.</a></li><li><a href=\"https://rlafuente.com/posts/2026-10-11-agent-evals-5-the-environment-is-the-eval\" rel=\"nofollow ugc noopener\">The environment is the eval.</a></li></ul>\n<p>Seven more parts are outlined on the <a href=\"https://rlafuente.com/agent-evals#outline\" rel=\"nofollow ugc noopener\">series page</a>.</p>\n<h2 id=\"references\">References</h2>\n<p>Anthropic (2026). <a href=\"https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\" rel=\"nofollow ugc noopener\">Demystifying evals for AI agents</a>. Anthropic Engineering.<br />\nChristiano, P. F. et al. (2017). <a href=\"https://arxiv.org/abs/1706.03741\" rel=\"nofollow ugc noopener\">Deep Reinforcement Learning from Human Preferences</a>. NeurIPS.<br />\nOuyang, L. et al. (2022). <a href=\"https://arxiv.org/abs/2203.02155\" rel=\"nofollow ugc noopener\">Training language models to follow instructions with human feedback</a>. NeurIPS.<br />\nRafailov, R. et al. (2023). <a href=\"https://arxiv.org/abs/2305.18290\" rel=\"nofollow ugc noopener\">Direct Preference Optimization: Your Language Model is Secretly a Reward Model</a>. NeurIPS.<br />\nLambert, N. et al. (2024). <a href=\"https://arxiv.org/abs/2411.15124\" rel=\"nofollow ugc noopener\">Tülu 3: Pushing Frontiers in Open Language Model Post-Training</a>.<br />\nDeepSeek-AI (2025). <a href=\"https://arxiv.org/abs/2501.12948\" rel=\"nofollow ugc noopener\">DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning</a>.<br />\nGao, L., Schulman, J., and Hilton, J. (2022). <a href=\"https://arxiv.org/abs/2210.10760\" rel=\"nofollow ugc noopener\">Scaling Laws for Reward Model Overoptimization</a>. ICML 2023.<br />\nShao, R. et al. (2025). <a href=\"https://arxiv.org/abs/2506.10947\" rel=\"nofollow ugc noopener\">Spurious Rewards: Rethinking Training Signals in RLVR</a>.<br />\nPryzant, R. et al. (2023). <a href=\"https://arxiv.org/abs/2305.03495\" rel=\"nofollow ugc noopener\">Automatic Prompt Optimization with &quot;Gradient Descent&quot; and Beam Search</a>. EMNLP.<br />\nYang, C. et al. (2023). <a href=\"https://arxiv.org/abs/2309.03409\" rel=\"nofollow ugc noopener\">Large Language Models as Optimizers</a>. ICLR 2024.<br />\nKhattab, O. et al. (2023). <a href=\"https://arxiv.org/abs/2310.03714\" rel=\"nofollow ugc noopener\">DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines</a>. ICLR 2024.<br />\nYuksekgonul, M. et al. (2024). <a href=\"https://arxiv.org/abs/2406.07496\" rel=\"nofollow ugc noopener\">TextGrad: Automatic &quot;Differentiation&quot; via Text</a>.<br />\nShinn, N. et al. (2023). <a href=\"https://arxiv.org/abs/2303.11366\" rel=\"nofollow ugc noopener\">Reflexion: Language Agents with Verbal Reinforcement Learning</a>. NeurIPS.<br />\nYao, S. et al. (2022). <a href=\"https://arxiv.org/abs/2210.03629\" rel=\"nofollow ugc noopener\">ReAct: Synergizing Reasoning and Acting in Language Models</a>. ICLR 2023.<br />\nYang, J. et al. (2024). <a href=\"https://arxiv.org/abs/2405.15793\" rel=\"nofollow ugc noopener\">SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering</a>. NeurIPS.<br />\nKapoor, S. et al. (2024). <a href=\"https://arxiv.org/abs/2407.01502\" rel=\"nofollow ugc noopener\">AI Agents That Matter</a>.<br />\nKapoor, S. et al. (2025). <a href=\"https://arxiv.org/abs/2510.11977\" rel=\"nofollow ugc noopener\">Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation</a>.<br />\nAnthropic (2024). <a href=\"https://modelcontextprotocol.io/\" rel=\"nofollow ugc noopener\">Model Context Protocol</a>. Specification and documentation.<br />\nPan, J. et al. (2024). <a href=\"https://arxiv.org/abs/2412.21139\" rel=\"nofollow ugc noopener\">Training Software Engineering Agents and Verifiers with SWE-Gym</a>.<br />\nYang, J. et al. (2025). <a href=\"https://arxiv.org/abs/2504.21798\" rel=\"nofollow ugc noopener\">SWE-smith: Scaling Data for Software Engineering Agents</a>.</p>","headings":[{"level":1,"text":"Two Hills: Weights and Harnesses","id":"two-hills-weights-and-harnesses"},{"level":2,"text":"The weights hill has a gradient","id":"the-weights-hill-has-a-gradient"},{"level":2,"text":"The harness hill has no gradient","id":"the-harness-hill-has-no-gradient"},{"level":2,"text":"Why this hill is new","id":"why-this-hill-is-new"},{"level":2,"text":"Who is on which hill","id":"who-is-on-which-hill"},{"level":2,"text":"References","id":"references"}]}}