This post is a hands-on introduction to reinforcement learning through training a search agent with group-relative policy optimization (GRPO). Search is a fun place to learn RL because it has so many levers, and each of them visibly changes how the model searches. It is also a domain where, in my experience, a well-designed reward function can influence how a model searches more effectively than system prompt changes or harness engineering.
\nOur approach is based loosely on the Harness-1 paper from Jiang et al. We use their training dataset and a much-simplified version of their search harness.
\nTraining is done using the Tinker API from Thinking Machines. Many thanks to the team for the training credits to support this work.
\nThe dataset
\nWe use the RL split of the Harness-1 dataset for all experiments. The dataset contains 3,453 queries over 37,113 SEC filings split into 2,115,106 chunks.
\nEach query includes a preamble followed by a question. The agent must both retrieve evidence for facts in the preamble and the answer to the question. In the canonical version of this task, retrieving the answer is more important than evidence for facts. However, for simplicity we will treat both the same.
\nTo keep initial ablations inexpensive, we start with just 256 training queries and 32 evaluation queries over a reduced corpus of 124,395 chunks. For our scaled runs, we expand to 1,024 training queries over 237,533 chunks.
\nThe harness
\nOur agent operates as a search subagent. Given a query, it is meant to retrieve all relevant docs. We keep the harness deliberately simple, only providing lexical search tools. The agent’s output is a curated set it builds up as it goes rather than a single submission at the end. We chose this because it works well for a fixed turn budget: since the set exists from the start, an episode that runs out of turns will will return something gradable.
\n| Tool | What it does |
\n|---|---|
\n| bm25_search | Keyword search over every chunk. Returns an id, title, score and snippet per hit, plus a running count of chunks seen and curated so far. |
\n| grep_corpus | A short Python regex scan across the corpus. Returns an id, title, and snippet per hit. |
\n| read_document | The title and text of one chunk. Only ids the agent has already seen in a result list can be read. |
\n| curate | Add chunk ids to the result set. |
\n| drop_curated | Remove ids from the result set. |
\n| finish | End the episode and return the set. |
\nThe system prompt is similarly plain. We name the tools and give a general strategy for search. We don’t try to encourage any specific search characteristics (e.g. high recall or high precision) so that the agent can discover the right behaviors during training.
\nRetrieval agent system prompt
\nYou are a retrieval subagent in a multi-agent system. Your specific role is to identify and retrieve the most relevant documents from a large corpus to help another agent answer questions. You do NOT answer questions yourself - you only find and retrieve relevant documents.
\nThe user message contains the query you need to find documents for.
\nAvailable Tools:
\nYour Process:
\nYour Thinking:
\nTactics to Consider:
\nOutput (IMPORTANT):
\nWe use gpt-oss-20b for all training runs. We chose this model because it is cheap to sample and train, allowing us to do many ablations.
\nQuick GRPO recap
\nWe would also typically use ratio clipping during the policy update to help stabilize training. However, to simplify training for the sake of pedagogy, we will train in a fully synchronous setting with zero training-inference mismatch (which Tinker provides by default) and only do one update per batch.
\nWe won’t go deeper into the math behind here, but there are many other great resources out there for understanding GRPO.
\nThe reward
\nEach task here consists of a query, a corpus, a set of facts, and a set of supporting documents for each fact. The agent has to curate a set of documents it determines to be relevant to those facts, and success is evaluated on how well this curated set covers the target facts.
\nA reward function is essentially a kind of evaluation scheme for a rollout. A few metrics common to search retrieval work naturally here.
\nRecall measures the fraction of relevant documents our agent has curated. Since our search problem is fact-based, this will be the fraction of relevant facts the curated chunks cover. Precision measures the fraction of curated documents that are actually relevant.
\nWe can’t use just recall or precision alone in our reward function. High recall is desirable in our problem space, but can be trivially maximized by curating every document in the corpus. This is useless for an end user! Why use a search agent at all? High precision can be achieved just as trivially by curating only a single correct chunk per query, which is useless in the other direction.
\nWe need a metric that punishes both forms of degenerate behavior. F1, the harmonic mean of precision and recall, is the usual choice here:
\nF1=P+R2PR
\nThis metric is only high when both precision and recall are high. This is a totally reasonable metric to choose for our reward function. But thinking about our problem space again… it’s likely that a missing document here costs far more to an end user than an extra, useless one. So, we might want to bias the reward towards recall. With this in mind, we will use F-beta as our reward function. This is a weighted version of F1 with a coefficient for setting how much recall matters more than precision:
\nFβ=β2P+R(1+β2)PR
\nWe set β = 4 for our reward function, which weighs recall sixteen times more than precision. I chose this number based on my prior experience and notes from the Harness-1 paper rather than a formal ablation. Under this metric, which we call F4, a set with recall 1.0 and precision 0.25 scores 0.85; under F1 the same set would score 0.40.
\nOne more thing we add to our reward function is a punishment for not curating anything—an episode that curates nothing at all gets a flat -0.2. This helps with differentiating between rollouts where the model never gets to curate anything and ones where it only curates incorrect documents.
\nPutting it together, the reward for a rollout with curated set C is
\nr(C)={−0.2F4if ∣C∣=0otherwise
\nInitial explorations
\nThe core idea behind reinforcement learning is to let models explore and reward them for behaviors that lead to good results. This means that RL works specifically by reinforcing behaviors the model already produces. If the model either always reaches for the same actions or always fails, then there is nothing for it to learn from.
\nThis is especially true in GRPO, where learning depends on variation in rewards within each group. If all rollouts receive the same reward, their advantages are zero and the group provides no gradient.
\nThis is why it can be helpful to gauge the RL-ability of a task before training. Best-of-N evals are a good proxy for this. Just sample N runs over the same task and report the highest score. A good candidate for RL is a model that has noticeably higher best-of-N scores than best-of-1 — this gap is the headroom RL can close by just sharpening the existing policy.
\nBelow, we sample gpt-oss-20b four times on each of the eval queries. All rollouts are viewable.
\nThe gap between best-of-4 and single trial F1 is large: best-of-4 F1 is 0.323 vs 0.195 for a single try. The F1 score is also precision-dominated. The model is likely being over-conservative in its curation.
\nClose inspection of the rollout supports this. On seven of the 32 queries, none of the four tries ever calls curate. The model stalls, exhausts all its turns, or runs out of context before curating a single document. Failures like these signal more of a harness problem than a search problem, and that is the kind of behavior reinforcement learning is good at fixing.
\nLearning rate sweep
\nLearning rate is a critical hyperparameter to get right. Just like in supervised fine-tuning, a bad learning rate can destabilize training. In reinforcement learning, a learning rate too high can cause the policy to collapse; one that is too small means the model never learns anything during training. And even when a run looks fine, I’ve found that further tuning the learning rate can often lead to incrementally better performance.
\nThat said, if you’re an experienced practitioner and know directionally what the right hyperparameters are, you can skip this step for initial experiments. My opinion is that once training works stably, tuning the learning rate only starts to matter when you’re trying to squeeze out extra performance from your data.
\nAll runs in this section and the next use the 256-query subset of our larger dataset, with 32 groups of 8 rollouts per step. Here are the results of a sweep over five different learning rates, each run for one epoch:
\ntrain reward
\ntrain reward | step | 5e-6 | 1e-5 | 5e-5 | 1e-4 | 5e-4 |
\n|---|---|---|---|---|---|
\n| 0 | -0.02 | -0.02 | -0.02 | -0.02 | -0.02 |
\n| 1 | -0.03 | -0.03 | -0.02 | -0.02 | 0.01 |
\n| 2 | -0.04 | -0.04 | -0.04 | -0.03 | -0.03 |
\n| 3 | -0.02 | -0.03 | -0.02 | 0.00 | 0.05 |
\n| 4 | 0.01 | -0.01 | 0.02 | 0.05 | 0.12 |
\n| 5 | -0.01 | 0.02 | 0.03 | 0.09 | 0.16 |
\n| 6 | -0.01 | 0.01 | 0.04 | 0.12 | 0.12 |
\n| 7 | -0.04 | -0.02 | 0.02 | 0.15 | 0.01 |
\n| 8 | -0.03 | -0.03 | 0.06 | 0.14 | 0.08 |
\nvalid train rollouts
\nvalid train rollouts | step | 5e-6 | 1e-5 | 5e-5 | 1e-4 | 5e-4 |
\n|---|---|---|---|---|---|
\n| 0 | 43% | 43% | 43% | 43% | 43% |
\n| 1 | 44% | 39% | 42% | 44% | 46% |
\n| 2 | 36% | 34% | 34% | 42% | 53% |
\n| 3 | 40% | 39% | 39% | 45% | 73% |
\n| 4 | 48% | 45% | 53% | 54% | 91% |
\n| 5 | 45% | 47% | 51% | 66% | 100% |
\n| 6 | 40% | 43% | 52% | 63% | 100% |
\n| 7 | 37% | 41% | 55% | 79% | 75% |
\n| 8 | 40% | 38% | 63% | 88% | 100% |
\neval f1
\neval f1 | step | 5e-6 | 1e-5 | 5e-5 | 1e-4 | 5e-4 |
\n|---|---|---|---|---|---|
\n| 0 | 0.19 | 0.19 | 0.19 | 0.19 | 0.19 |
\n| 4 | 0.18 | 0.21 | 0.22 | 0.20 | 0.12 |
\n| 8 | 0.16 | 0.14 | 0.16 | 0.19 | 0.03 |
\nTraining results over one epoch with different learning rates. The valid rollouts chart shows the percentage of training rollouts that curate at least one doc.
\nWe can see in here the effects of both a too large and a too small learning rate:
\nSince lr=1e-4 performs the best in the sweep, we’ll use it for the rest of our experiments.
\nIf you want a better idea of what the training collapse for lr=5e-4 looks like, expand the explorer below to see its eval rollouts at steps 0 and 8.
\nBatch size ablation
\nConventional wisdom when training with GRPO is to use large batch sizes, where batch size is groups per step x group size. To study the effects of batch size on training here, we fix group size at 8 and ablate over groups per step.
\nThe charts below are plotted against prompts seen rather than total updates, since all runs have a different total number of steps.
\ntrain reward
\ntrain reward | step | 8 groups/step | 16 groups/step | 32 groups/step | 64 groups/step |
\n|---|---|---|---|---|
\n| 0 | -0.03 | -0.03 | -0.03 | -0.03 |
\n| 8 | -0.01 | | | |
\n| 16 | 0.04 | 0.02 | | |
\n| 24 | 0.04 | | | |
\n| 32 | -0.06 | -0.07 | -0.01 | |
\n| 40 | -0.06 | | | |
\n| 48 | -0.00 | -0.06 | | |
\n| 56 | 0.04 | | | |
\n| 64 | 0.25 | 0.04 | -0.04 | -0.05 |
\n| 72 | 0.03 | | | |
\n| 80 | 0.06 | -0.01 | | |
\n| 88 | 0.11 | | | |
\n| 96 | 0.14 | 0.09 | -0.01 | |
\n| 104 | 0.03 | | | |
\n| 112 | 0.07 | 0.04 | | |
\n| 120 | 0.13 | | | |
\n| 128 | 0.21 | 0.20 | 0.01 | -0.01 |
\n| 136 | 0.24 | | | |
\n| 144 | 0.16 | 0.21 | | |
\n| 152 | 0.17 | | | |
\n| 160 | 0.02 | 0.10 | 0.10 | |
\n| 168 | 0.10 | | | |
\n| 176 | 0.13 | 0.14 | | |
\n| 184 | 0.19 | | | |
\n| 192 | 0.09 | 0.14 | 0.08 | 0.07 |
\n| 200 | 0.16 | | | |
\n| 208 | 0.14 | 0.16 | | |
\n| 216 | 0.07 | | | |
\n| 224 | 0.08 | 0.11 | 0.12 | |
\n| 232 | 0.13 | | | |
\n| 240 | 0.06 | 0.07 | | |
\n| 248 | 0.05 | | | |
\n| 256 | 0.17 | 0.12 | 0.16 | 0.05 |
\nvalid train rollouts
\nvalid train rollouts | step | 8 groups/step | 16 groups/step | 32 groups/step | 64 groups/step |
\n|---|---|---|---|---|
\n| 0 | 41% | 41% | 41% | 41% |
\n| 8 | 36% | | | |
\n| 16 | 55% | 45% | | |
\n| 24 | 60% | | | |
\n| 32 | 39% | 43% | 44% | |
\n| 40 | 66% | | | |
\n| 48 | 72% | 48% | | |
\n| 56 | 85% | | | |
\n| 64 | 98% | 61% | 42% | 40% |
\n| 72 | 92% | | | |
\n| 80 | 97% | 62% | | |
\n| 88 | 100% | | | |
\n| 96 | 98% | 72% | 45% | |
\n| 104 | 100% | | | |
\n| 112 | 100% | 75% | | |
\n| 120 | 100% | | | |
\n| 128 | 100% | 90% | 54% | 40% |
\n| 136 | 100% | | | |
\n| 144 | 100% | 94% | | |
\n| 152 | 100% | | | |
\n| 160 | 100% | 92% | 66% | |
\n| 168 | 100% | | | |
\n| 176 | 100% | 95% | | |
\n| 184 | 100% | | | |
\n| 192 | 100% | 94% | 63% | 51% |
\n| 200 | 100% | | | |
\n| 208 | 100% | 98% | | |
\n| 216 | 100% | | | |
\n| 224 | 98% | 95% | 79% | |
\n| 232 | 100% | | | |
\n| 240 | 100% | 97% | | |
\n| 248 | 100% | | | |
\n| 256 | 100% | 95% | 88% | 56% |
\neval f1
\neval f1 | step | 8 groups/step | 16 groups/step | 32 groups/step | 64 groups/step |
\n|---|---|---|---|---|
\n| 0 | 0.19 | 0.19 | 0.19 | 0.19 |
\n| 64 | 0.20 | 0.18 | 0.24 | 0.17 |
\n| 128 | 0.13 | 0.20 | 0.20 | 0.21 |
\n| 192 | 0.12 | 0.11 | 0.20 | 0.29 |
\n| 256 | 0.14 | 0.18 | 0.19 | 0.24 |
\nAt a glance, the eval results are in line with expectations: at the end of training, 64 groups/step has the highest eval F1, and each smaller batch size monotonically decreases in final eval F1. Larger batches also give smoother reward and eval curves.
\nThere’s an interesting contradiction in the charts, though. Although 64 groups/step ends with the highest eval score, it has the lowest train reward at the end of training. Also, 8 groups/step has the highest train reward, but the lowest F1 score.
\nThis has to do with our reward setup and training dynamics — the reward on a rollout that curates nothing is -0.2, which far outweighs the average reward on most valid rollouts. So, the fastest way to raise train reward early on is to just curate any document. Because 8 groups/step takes more updates than any other run, the model starts to produce valid rollouts sooner. The middle chart above shows this: as batch size decreases, the model reaches >75% valid train rollouts faster. The problem from there is that noisy updates (due to small batch size) keep the policy’s curation quality from ever meaningfully improving. What we’re seeing here is actually a form of reward hacking. The next section will explore this more.
\nSince 64 groups/step gives the most stable training and has the highest eval score, we will use it for our full training runs. We could technically increase groups/step further, but for the sake of budget we will stop here.
\nScaling up training
\nFor our next set of training runs, we will move to the larger dataset of 1,024 queries and 237,533 chunks. This is 4x the data samples and 2x the corpus size of the dataset we used for ablations. We train for 1.5 epochs, which at 64 groups/step is 24 total updates.
\nFull training configuration
\n| model |
\n| model | gpt-oss-20b |
\n| adapter | LoRA, rank 32 |
\n| optimisation |
\n| learning rate | 1e-4 |
\n| optimizer | Adam, β₁ 0.9, β₂ 0.95, ε 1e-8 |
\n| weight decay | 0 |
\n| KL penalty to reference | 0.0 |
\n| updates per batch | 1 |
\n| sampling temperature | 1.0 |
\n| harness |
\n| max turns | 40 |
\n| max new tokens per turn | 2,048 |
\n| context length limit | 30,720 tokens |
\n| curated set cap | 30 |
\nWe first look at training metrics over the full run. In the spirit of openness, every training rollout can be viewed in the explorer below.
\ntrain reward
\ntrain reward | step | train reward |
\n|---|---|
\n| 0 | -0.06 |
\n| 1 | 0.02 |
\n| 2 | 0.01 |
\n| 3 | 0.03 |
\n| 4 | 0.04 |
\n| 5 | 0.10 |
\n| 6 | 0.17 |
\n| 7 | 0.15 |
\n| 8 | 0.14 |
\n| 9 | 0.15 |
\n| 10 | 0.13 |
\n| 11 | 0.17 |
\n| 12 | 0.15 |
\n| 13 | 0.16 |
\n| 14 | 0.18 |
\n| 15 | 0.21 |
\n| 16 | 0.25 |
\n| 17 | 0.17 |
\n| 18 | 0.22 |
\n| 19 | 0.23 |
\n| 20 | 0.21 |
\n| 21 | 0.27 |
\n| 22 | 0.24 |
\n| 23 | 0.21 |
\nvalid rollouts
\nvalid rollouts | step | valid rollouts |
\n|---|---|
\n| 0 | 30% |
\n| 1 | 42% |
\n| 2 | 47% |
\n| 3 | 64% |
\n| 4 | 79% |
\n| 5 | 92% |
\n| 6 | 96% |
\n| 7 | 98% |
\n| 8 | 98% |
\n| 9 | 98% |
\n| 10 | 98% |
\n| 11 | 97% |
\n| 12 | 94% |
\n| 13 | 95% |
\n| 14 | 94% |
\n| 15 | 90% |
\n| 16 | 92% |
\n| 17 | 92% |
\n| 18 | 95% |
\n| 19 | 96% |
\n| 20 | 96% |
\n| 21 | 94% |
\n| 22 | 94% |
\n| 23 | 96% |
\nentropy (nats)
\nentropy (nats) | step | entropy (nats) |
\n|---|---|
\n| 0 | 1.1 |
\n| 1 | 1.0 |
\n| 2 | 1.0 |
\n| 3 | 1.0 |
\n| 4 | 1.0 |
\n| 5 | 0.9 |
\n| 6 | 0.8 |
\n| 7 | 0.8 |
\n| 8 | 0.8 |
\n| 9 | 0.8 |
\n| 10 | 0.8 |
\n| 11 | 0.8 |
\n| 12 | 0.8 |
\n| 13 | 0.8 |
\n| 14 | 0.9 |
\n| 15 | 0.9 |
\n| 16 | 0.9 |
\n| 17 | 0.9 |
\n| 18 | 0.9 |
\n| 19 | 0.9 |
\n| 20 | 0.9 |
\n| 21 | 0.9 |
\n| 22 | 0.9 |
\n| 23 | 0.8 |
\nTrain reward moves the way we’d expect it to here: up rapidly at first, and then slowing down as training matures. The training shape also mirrors what we see in ablations: the reward first teaches the model to curate. After that, reward moves around as the model starts to learn how to curate well instead of just how to curate. You can think of the -0.2 floor on no curations as jump-starting curation behavior.
\nEntropy is another useful metric to watch during reinforcement learning. Entropy is measured in nats here. One nat is 1 / ln 2, or about 1.44 bits. It measures how uncertain the policy’s output distribution is over the course of a rollout. High entropy means at several points in the rollout, the model is considering several possible actions (next tokens). As we’ve discussed earlier, this is good for RL because it means the model will explore more and hopefully stumble into behaviors that get rewarded. Low entropy isn’t inherently bad, but a rapid drop during training is a sign the model is becoming overconfident or committing to a single set of behaviors. This is known as entropy collapse and is particularly bad for GRPO — a policy that always takes the same actions produces groups with no variance, and groups with no variance produce no gradient.
\nDuring healthy training, entropy typically decreases gradually as reward improves. That is exactly what we see here: entropy starts at 1.1 nats, gradually goes to ~0.8 a quarter of the way through training, and then stays there.
\nLet’s see how our policy performs on held-out data now.
\neval f1
\neval f1 | step | eval f1 |
\n|---|---|
\n| 0 | 0.19 |
\n| 2 | 0.22 |
\n| 4 | 0.17 |
\n| 6 | 0.19 |
\n| 8 | 0.21 |
\n| 10 | 0.17 |
\n| 12 | 0.16 |
\n| 14 | 0.24 |
\n| 16 | 0.23 |
\n| 18 | 0.29 |
\n| 20 | 0.31 |
\n| 22 | 0.28 |
\n| 24 | 0.33 |
\neval precision
\neval precision | step | eval precision |
\n|---|---|
\n| 0 | 0.30 |
\n| 2 | 0.38 |
\n| 4 | 0.30 |
\n| 6 | 0.29 |
\n| 8 | 0.32 |
\n| 10 | 0.27 |
\n| 12 | 0.17 |
\n| 14 | 0.35 |
\n| 16 | 0.30 |
\n| 18 | 0.36 |
\n| 20 | 0.40 |
\n| 22 | 0.36 |
\n| 24 | 0.40 |
\neval recall
\neval recall | step | eval recall |
\n|---|---|
\n| 0 | 0.16 |
\n| 2 | 0.17 |
\n| 4 | 0.13 |
\n| 6 | 0.16 |
\n| 8 | 0.17 |
\n| 10 | 0.14 |
\n| 12 | 0.18 |
\n| 14 | 0.21 |
\n| 16 | 0.20 |
\n| 18 | 0.27 |
\n| 20 | 0.28 |
\n| 22 | 0.26 |
\n| 24 | 0.32 |
\nProgress is uneven early on as the model learns our harness, and eval performance only starts to improve around halfway through training. Mean eval F1 rises from 0.193 at the start to 0.332 at update 24, driven almost entirely by recall, which doubles from 0.16 to 0.32. Precision goes up as well, moving from 0.31 to 0.40. This is a good result because it means we are getting an all around improvement. That is, we aren’t trading off precision for recall in any way.
\nThe model performance doesn’t come at the cost of test-time compute, either. As the following charts show, the turn count actually falls slightly over training and output tokens go down, which means our final model is finding more relevant docs with less searching.
\nturns per episode
\nturns per episode | step | turns per episode |
\n|---|---|
\n| 0 | 18 |
\n| 1 | 18 |
\n| 2 | 18 |
\n| 3 | 16 |
\n| 4 | 15 |
\n| 5 | 13 |
\n| 6 | 12 |
\n| 7 | 12 |
\n| 8 | 12 |
\n| 9 | 12 |
\n| 10 | 12 |
\n| 11 | 12 |
\n| 12 | 14 |
\n| 13 | 14 |
\n| 14 | 16 |
\n| 15 | 15 |
\n| 16 | 16 |
\n| 17 | 16 |
\n| 18 | 15 |
\n| 19 | 15 |
\n| 20 | 14 |
\n| 21 | 14 |
\n| 22 | 16 |
\n| 23 | 17 |
\noutput tokens per episode
\noutput tokens per episode | step | output tokens per episode |
\n|---|---|
\n| 0 | 3.998k |
\n| 1 | 3.47k |
\n| 2 | 3.05k |
\n| 3 | 2.621k |
\n| 4 | 2.202k |
\n| 5 | 1.754k |
\n| 6 | 1.408k |
\n| 7 | 1.438k |
\n| 8 | 1.414k |
\n| 9 | 1.497k |
\n| 10 | 1.698k |
\n| 11 | 1.709k |
\n| 12 | 1.972k |
\n| 13 | 1.954k |
\n| 14 | 2.328k |
\n| 15 | 2.461k |
\n| 16 | 2.527k |
\n| 17 | 2.898k |
\n| 18 | 2.693k |
\n| 19 | 2.654k |
\n| 20 | 2.673k |
\n| 21 | 2.649k |
\n| 22 | 2.825k |
\n| 23 | 2.74k |
\nSo, we have a full training run that works and pushes eval scores noticeably higher. This is a great result! In all likelihood, we haven’t saturated our training setup, either—train reward is still steadily going up around update 24. We’d probably continue to see eval gains by scaling up the dataset or running for another epoch…
\nBut that’s a boring next step to explore here. There are many other fun ways to squeeze out performance from our dataset.
\nAside: What happened with 8 groups/step?
\nIn our ablations, we saw that the 8 groups/step run quickly learned to curate, but this failed to translate over meaningfully into eval improvements. We extended that run over the larger dataset to see if more training steps would help. The charts below show the results of training over 8 groups/step vs 64 groups/step over the same first 384 queries of the larger dataset.
\ntrain reward
\ntrain reward | step | 64 groups/step | 8 groups/step |
\n|---|---|---|
\n| 0 | -0.06 | 0.06 |
\n| 8 | | 0.03 |
\n| 16 | | 0.06 |
\n| 24 | | 0.03 |
\n| 32 | | 0.13 |
\n| 40 | | 0.07 |
\n| 48 | | 0.23 |
\n| 56 | | 0.16 |
\n| 64 | 0.02 | 0.15 |
\n| 72 | | 0.21 |
\n| 80 | | 0.30 |
\n| 88 | | 0.18 |
\n| 96 | | 0.27 |
\n| 104 | | 0.12 |
\n| 112 | | 0.09 |
\n| 120 | | 0.20 |
\n| 128 | 0.01 | 0.11 |
\n| 136 | | 0.12 |
\n| 144 | | 0.13 |
\n| 152 | | 0.06 |
\n| 160 | | 0.10 |
\n| 168 | | 0.15 |
\n| 176 | | 0.08 |
\n| 184 | | 0.17 |
\n| 192 | 0.03 | 0.09 |
\n| 200 | | 0.14 |
\n| 208 | | 0.09 |
\n| 216 | | 0.13 |
\n| 224 | | 0.09 |
\n| 232 | | 0.09 |
\n| 240 | | 0.21 |
\n| 248 | | 0.30 |
\n| 256 | 0.04 | 0.10 |
\n| 264 | | 0.18 |
\n| 272 | | 0.23 |
\n| 280 | | 0.03 |
\n| 288 | | 0.10 |
\n| 296 | | 0.14 |
\n| 304 | | 0.25 |
\n| 312 | | 0.19 |
\n| 320 | 0.10 | 0.19 |
\n| 328 | | 0.25 |
\n| 336 | | 0.36 |
\n| 344 | | 0.13 |
\n| 352 | | 0.15 |
\n| 360 | | 0.08 |
\n| 368 | | 0.21 |
\n| 376 | | 0.16 |
\n| 384 | 0.17 | |
\ncurated documents
\ncurated documents | step | 64 groups/step | 8 groups/step |
\n|---|---|---|
\n| 0 | 1 | 1 |
\n| 8 | | 1 |
\n| 16 | | 1 |
\n| 24 | | 1 |
\n| 32 | | 1 |
\n| 40 | | 1 |
\n| 48 | | 2 |
\n| 56 | | 3 |
\n| 64 | 1 | 2 |
\n| 72 | | 3 |
\n| 80 | | 4 |
\n| 88 | | 4 |
\n| 96 | | 5 |
\n| 104 | | 5 |
\n| 112 | | 6 |
\n| 120 | | 7 |
\n| 128 | 1 | 6 |
\n| 136 | | 9 |
\n| 144 | | 8 |
\n| 152 | | 7 |
\n| 160 | | 7 |
\n| 168 | | 7 |
\n| 176 | | 7 |
\n| 184 | | 7 |
\n| 192 | 1 | 9 |
\n| 200 | | 12 |
\n| 208 | | 9 |
\n| 216 | | 9 |
\n| 224 | | 11 |
\n| 232 | | 11 |
\n| 240 | | 9 |
\n| 248 | | 13 |
\n| 256 | 1 | 18 |
\n| 264 | | 13 |
\n| 272 | | 12 |
\n| 280 | | 17 |
\n| 288 | | 16 |
\n| 296 | | 17 |
\n| 304 | | 21 |
\n| 312 | | 21 |
\n| 320 | 2 | 24 |
\n| 328 | | 22 |
\n| 336 | | 23 |
\n| 344 | | 24 |
\n| 352 | | 24 |
\n| 360 | | 24 |
\n| 368 | | 25 |
\n| 376 | | 25 |
\n| 384 | 2 | |
\nentropy (nats)
\nentropy (nats) | step | 64 groups/step | 8 groups/step |
\n|---|---|---|
\n| 0 | 1.1 | 1.1 |
\n| 8 | | 1.1 |
\n| 16 | | 1.0 |
\n| 24 | | 0.9 |
\n| 32 | | 0.9 |
\n| 40 | | 0.9 |
\n| 48 | | 0.8 |
\n| 56 | | 0.7 |
\n| 64 | 1.0 | 0.8 |
\n| 72 | | 0.7 |
\n| 80 | | 0.7 |
\n| 88 | | 0.7 |
\n| 96 | | 0.7 |
\n| 104 | | 0.7 |
\n| 112 | | 0.7 |
\n| 120 | | 0.7 |
\n| 128 | 1.0 | 0.8 |
\n| 136 | | 0.7 |
\n| 144 | | 0.8 |
\n| 152 | | 0.7 |
\n| 160 | | 0.7 |
\n| 168 | | 0.7 |
\n| 176 | | 0.7 |
\n| 184 | | 0.8 |
\n| 192 | 1.0 | 0.7 |
\n| 200 | | 0.8 |
\n| 208 | | 0.7 |
\n| 216 | | 0.7 |
\n| 224 | | 0.7 |
\n| 232 | | 0.6 |
\n| 240 | | 0.6 |
\n| 248 | | 0.7 |
\n| 256 | 1.0 | 0.5 |
\n| 264 | | 0.6 |
\n| 272 | | 0.6 |
\n| 280 | | 0.5 |
\n| 288 | | 0.5 |
\n| 296 | | 0.5 |
\n| 304 | | 0.4 |
\n| 312 | | 0.5 |
\n| 320 | 0.9 | 0.4 |
\n| 328 | | 0.4 |
\n| 336 | | 0.4 |
\n| 344 | | 0.4 |
\n| 352 | | 0.3 |
\n| 360 | | 0.3 |
\n| 368 | | 0.3 |
\n| 376 | | 0.3 |
\n| 384 | 0.8 | |
\neval f1
\neval f1 | step | 64 groups/step | 8 groups/step |
\n|---|---|---|
\n| 0 | 0.19 | 0.19 |
\n| 128 | 0.25 | 0.12 |
\n| 256 | 0.15 | 0.16 |
\n| 384 | 0.17 | 0.16 |
\neval precision
\neval precision | step | 64 groups/step | 8 groups/step |
\n|---|---|---|
\n| 0 | 0.30 | 0.30 |
\n| 128 | 0.38 | 0.11 |
\n| 256 | 0.30 | 0.14 |
\n| 384 | 0.29 | 0.13 |
\neval recall
\neval recall | step | 64 groups/step | 8 groups/step |
\n|---|---|---|
\n| 0 | 0.16 | 0.16 |
\n| 128 | 0.17 | 0.14 |
\n| 256 | 0.13 | 0.23 |
\n| 384 | 0.16 | 0.29 |
\nThe 8 groups/step run reaches higher training reward, but it gets there by learning to just curate everything — although eval recall rises from 0.16 to 0.29, precision drops from 0.31 to 0.13. Entropy also falls from 1.1 to 0.34 nats over the course of training. This is a clear sign of entropy collapse. It’s likely that learning will outright stop if we were to continue training.
\nPart of this is our reward choice. We chose β = 4 to push the model towards recall over precision, and F4 weights recall 16x. Small-batch updates are also noisier, so once the policy stumbles onto the “curate every document you see” strategy, there is little pushing the model back to something more reasonable.
\nOne way to mitigate the reward bias here would be to anneal β over the course of training, as is done in Chroma Context-1.
\nReward shaping
\nEvery run so far has used the plain F4 score as the reward. It worked, in the sense that eval scores went up. But reading the rollouts directly and exploring a different set of training metrics shows two other ways we can improve the policy.
\ntrajectory vs curated recall
\ntrajectory vs curated recall | step | trajectory recall | curated recall |
\n|---|---|---|
\n| 0 | 0.46 | 0.12 |
\n| 1 | 0.51 | 0.16 |
\n| 2 | 0.40 | 0.13 |
\n| 3 | 0.34 | 0.11 |
\n| 4 | 0.29 | 0.09 |
\n| 5 | 0.32 | 0.14 |
\n| 6 | 0.39 | 0.22 |
\n| 7 | 0.38 | 0.24 |
\n| 8 | 0.36 | 0.23 |
\n| 9 | 0.33 | 0.21 |
\n| 10 | 0.26 | 0.19 |
\n| 11 | 0.33 | 0.24 |
\n| 12 | 0.29 | 0.21 |
\n| 13 | 0.30 | 0.21 |
\n| 14 | 0.27 | 0.18 |
\n| 15 | 0.38 | 0.26 |
\n| 16 | 0.40 | 0.28 |
\n| 17 | 0.30 | 0.20 |
\n| 18 | 0.39 | 0.28 |
\n| 19 | 0.37 | 0.28 |
\n| 20 | 0.36 | 0.27 |
\n| 21 | 0.41 | 0.32 |
\n| 22 | 0.39 | 0.29 |
\n| 23 | 0.36 | 0.27 |
\nTrajectory recall measures how many relevant chunks the model encountered while searching. tool-call format drift
\ntool-call format drift | step | of calls | of episodes |
\n|---|---|---|
\n| 0 | 27.3% | 37.9% |
\n| 1 | 13.7% | 29.3% |
\n| 2 | 15% | 26.4% |
\n| 3 | 9.1% | 16.2% |
\n| 4 | 11.5% | 18.9% |
\n| 5 | 12.4% | 18.8% |
\n| 6 | 14.8% | 17.8% |
\n| 7 | 25% | 32.2% |
\n| 8 | 21.2% | 32.2% |
\n| 9 | 19.7% | 32.4% |
\n| 10 | 14.6% | 33.6% |
\n| 11 | 26.8% | 41.6% |
\n| 12 | 28.3% | 55.9% |
\n| 13 | 47.2% | 67.2% |
\n| 14 | 49.6% | 74.2% |
\n| 15 | 62.6% | 79.9% |
\n| 16 | 53.8% | 78.3% |
\n| 17 | 58.5% | 89.3% |
\n| 18 | 67.3% | 90.2% |
\n| 19 | 67.9% | 92.6% |
\n| 20 | 63% | 92.4% |
\n| 21 | 62.6% | 91% |
\n| 22 | 67.7% | 96.7% |
\n| 23 | 75% | 98.8% |
\nMeasures % of tool calls or episodes with incorrect harmony format.
\nThese observations suggest two ways we can improve the model further: reward the model for encountering more evidence during search, and punish it for bad tool calls.
\nRewarding discovery
\nThe trajectory recall chart tells us that the model either isn’t searching well or isn’t searching enough. There are two ways we can influence this behavior.
\nFirst, we can reward the behavior directly: a bonus per search or grep call, or for more varied queries, and hope that together with the F4 reward, this leads to better coverage. However, a bonus just for searching is easy to hack. The model could just spam irrelevant calls and collect the bonus without learning any performance-improving behavior. We’d need to add a turn penalty to hold it back. This adds a second knob to tune and complicates the reward.
\nAnother route is to add an explicit trajectory recall term to our reward. This is a better route because we’re rewarding the outcome we want rather than the activity it takes to get there. As our charts show, trajectory recall is directionally aligned with curated recall, and it’s also harder to game — extra searches don’t earn anything unless they are helpful. This is the route we’ll experiment with.
\nConcretely, we add a term for the fraction of the query’s facts that have a supporting chunk anywhere in the trajectory, curated or not:
\nr(C)={−0.2F4+0.2⋅Rtrajif ∣C∣=0otherwise
\nThe explorer below shows the impact of different reward shapes on a group. Seven of the eight rollouts score zero F4 and are indistinguishable under the older reward, but separable under either bonus.
\nApart from the reward, every setting in this run matches the full run above.
\neval f1
\neval f1 | step | w/o trajectory reward | w trajectory reward |
\n|---|---|---|
\n| 0 | 0.19 | 0.19 |
\n| 2 | 0.22 | 0.21 |
\n| 4 | 0.17 | 0.21 |
\n| 6 | 0.19 | 0.18 |
\n| 8 | 0.21 | 0.18 |
\n| 10 | 0.17 | 0.27 |
\n| 12 | 0.16 | 0.27 |
\n| 14 | 0.24 | 0.25 |
\n| 16 | 0.23 | 0.28 |
\n| 18 | 0.29 | 0.27 |
\n| 20 | 0.31 | 0.32 |
\n| 22 | 0.28 | 0.36 |
\n| 24 | 0.33 | 0.33 |
\neval curated recall
\neval curated recall | step | w/o trajectory reward | w trajectory reward |
\n|---|---|---|
\n| 0 | 0.16 | 0.16 |
\n| 2 | 0.17 | 0.15 |
\n| 4 | 0.13 | 0.16 |
\n| 6 | 0.16 | 0.14 |
\n| 8 | 0.17 | 0.14 |
\n| 10 | 0.14 | 0.22 |
\n| 12 | 0.18 | 0.23 |
\n| 14 | 0.21 | 0.20 |
\n| 16 | 0.20 | 0.24 |
\n| 18 | 0.27 | 0.24 |
\n| 20 | 0.28 | 0.29 |
\n| 22 | 0.26 | 0.34 |
\n| 24 | 0.32 | 0.30 |
\neval trajectory recall
\neval trajectory recall | step | w/o trajectory reward | w trajectory reward |
\n|---|---|---|
\n| 0 | 0.43 | 0.43 |
\n| 2 | 0.35 | 0.37 |
\n| 4 | 0.30 | 0.30 |
\n| 6 | 0.29 | 0.26 |
\n| 8 | 0.28 | 0.26 |
\n| 10 | 0.22 | 0.33 |
\n| 12 | 0.22 | 0.33 |
\n| 14 | 0.28 | 0.36 |
\n| 16 | 0.30 | 0.38 |
\n| 18 | 0.36 | 0.47 |
\n| 20 | 0.37 | 0.48 |
\n| 22 | 0.33 | 0.53 |
\n| 24 | 0.44 | 0.49 |
\nEval is run 2x, and plotted points are the averages. Reward plots omitted because they are noncomparable.
\nTrajectory recall is clearly better here: after a dip around step 8, the shaped run recovers sooner and stays above the old, plain-F4 run. Eval F1 and curated recall also trend above the old run and even peak higher. Unfortunately, this doesn’t translate directly to a better final score under the same training budget.
\nHarmony, the format gpt-oss-20b uses, has a specific header for tool calls. At the start of training, the model gets it wrong on about a quarter of calls.
\nBecause we use a lenient parse that accepts these slightly off-form calls, the model still gets rewarded whenever the episode they’re in happens to get a good score. This habit ends up compounding. By the final step of training, 99% of episodes contain at least one bad tool call. The model isn’t being told, in any way that reaches the gradient, that those calls are wrong, so it ends up learning that those calls are acceptable.
\nThe following experiment fixes that directly. Every setting is the same as our full, plain F4 run, except for a slight change to the reward: subtract 0.1 if any tool call in the episode has an off-form header. If two rollouts in a group reach the same F4 but one uses a bad header, the clean one now gets more advantage.
\nf1
\nf1 | step | w/o format penalty | w format penalty |
\n|---|---|---|
\n| 0 | 0.19 | 0.19 |
\n| 2 | 0.22 | 0.20 |
\n| 4 | 0.17 | 0.27 |
\n| 6 | 0.19 | 0.21 |
\n| 8 | 0.21 | 0.24 |
\n| 10 | 0.17 | 0.20 |
\n| 12 | 0.16 | 0.25 |
\n| 14 | 0.24 | 0.27 |
\n| 16 | 0.23 | 0.34 |
\n| 18 | 0.29 | 0.33 |
\n| 20 | 0.31 | 0.39 |
\n| 22 | 0.28 | 0.36 |
\n| 24 | 0.33 | 0.36 |
\nrecall
\nrecall | step | w/o format penalty | w format penalty |
\n|---|---|---|
\n| 0 | 0.16 | 0.16 |
\n| 2 | 0.17 | 0.16 |
\n| 4 | 0.13 | 0.20 |
\n| 6 | 0.16 | 0.17 |
\n| 8 | 0.17 | 0.20 |
\n| 10 | 0.14 | 0.20 |
\n| 12 | 0.18 | 0.26 |
\n| 14 | 0.21 | 0.26 |
\n| 16 | 0.20 | 0.36 |
\n| 18 | 0.27 | 0.35 |
\n| 20 | 0.28 | 0.43 |
\n| 22 | 0.26 | 0.42 |
\n| 24 | 0.32 | 0.43 |
\noff-form tool headers
\noff-form tool headers | step | w/o format penalty (episodes) | w format penalty (episodes) | w/o format penalty (calls) | w format penalty (calls) |
\n|---|---|---|---|---|
\n| 0 | 37.9% | 39.8% | 27.3% | 27.8% |
\n| 1 | 29.3% | 23.4% | 13.7% | 10.3% |
\n| 2 | 26.4% | 21.7% | 15% | 6.1% |
\n| 3 | 16.2% | 9.2% | 9.1% | 2.7% |
\n| 4 | 18.9% | 6.8% | 11.5% | 2.3% |
\n| 5 | 18.8% | 3.9% | 12.4% | 0.7% |
\n| 6 | 17.8% | 3.5% | 14.8% | 1% |
\n| 7 | 32.2% | 7% | 25% | 2.1% |
\n| 8 | 32.2% | 4.9% | 21.2% | 0.5% |
\n| 9 | 32.4% | 9.6% | 19.7% | 2.8% |
\n| 10 | 33.6% | 12.7% | 14.6% | 3.6% |
\n| 11 | 41.6% | 8.6% | 26.8% | 1.1% |
\n| 12 | 55.9% | 8% | 28.3% | 0.8% |
\n| 13 | 67.2% | 7.2% | 47.2% | 0.6% |
\n| 14 | 74.2% | 8.2% | 49.6% | 0.5% |
\n| 15 | 79.9% | 8.6% | 62.6% | 0.4% |
\n| 16 | 78.3% | 5.9% | 53.8% | 0.4% |
\n| 17 | 89.3% | 5.1% | 58.5% | 0.2% |
\n| 18 | 90.2% | 4.9% | 67.3% | 0.2% |
\n| 19 | 92.6% | 3.1% | 67.9% | 0.1% |
\n| 20 | 92.4% | 3.1% | 63% | 0.1% |
\n| 21 | 91% | 4.3% | 62.6% | 0.2% |
\n| 22 | 96.7% | 4.7% | 67.7% | 0.2% |
\n| 23 | 98.8% | 8.2% | 75% | 0.4% |
\nturns per episode
\nturns per episode | step | w/o format penalty | w format penalty |
\n|---|---|---|
\n| 0 | 18 | 18 |
\n| 1 | 18 | 17 |
\n| 2 | 18 | 17 |
\n| 3 | 16 | 13 |
\n| 4 | 15 | 12 |
\n| 5 | 13 | 12 |
\n| 6 | 12 | 12 |
\n| 7 | 12 | 13 |
\n| 8 | 12 | 15 |
\n| 9 | 12 | 14 |
\n| 10 | 12 | 15 |
\n| 11 | 12 | 16 |
\n| 12 | 14 | 20 |
\n| 13 | 14 | 20 |
\n| 14 | 16 | 21 |
\n| 15 | 15 | 23 |
\n| 16 | 16 | 23 |
\n| 17 | 16 | 25 |
\n| 18 | 15 | 26 |
\n| 19 | 15 | 27 |
\n| 20 | 14 | 27 |
\n| 21 | 14 | 28 |
\n| 22 | 16 | 30 |
\n| 23 | 17 | 30 |
\nIn the tool headers chart, solid lines show the share of episodes with off-form calls; dashed lines show the share of individual calls.
\nThe extra penalty works, fast. Off-form tool calls fall below 1% within a few steps and stay there for the rest of training. Eval metrics and recall also improve against the baseline. Held-out F1 ends at 0.36 vs 0.33, and recall at 0.43 vs 0.32.
\nThis result surprised me. I expected the penalty to fix tool call formatting, but not necessarily retrieval quality. Yet the model starts taking longer trajectories (does more searches) and achieves higher recall. I don’t have a concrete explanation for this. One possibility is that the penalty changes which search behaviors get reinforced, and well-formed tool calls are happen to be correlated with better search behaviors. Another is that the model just has more coherent long-context understanding when all of its preceding tool calls are well-formed.
\nEither hypothesis would require more training runs to test, but I’ll stop here for the sake of my training budget. What we should take away from this is that a seemingly narrow reward change can sometimes lead to big shifts in the trained policy. We need to be careful and intentional when working on reward functions.
\nSome closing thoughts
\nWe started with a 20B model that struggled to properly execute over our harness, then ran ablations and experiments to drive its eval F1 score from 0.19 up to a peak of 0.39.
\nThere’s still a lot more we could do here to improve performance. For one, we only trained on a fraction of the total SEC Harness-1 dataset. We could expand the training set or test more learning-rate and batch-size combinations. Tuning one hyperparameter at a time was the right call under our specific budget, but we could have carried over a few configurations to the full run.
\n