We have seen this play out before

In the early days of language modeling we had exactly this problem. A big pretrained model like GPT-2 or GPT-3 was fluent, knowledgeable, and completely unreliable. If you ask GPT-2 “How do I roast a whole chicken?”, you might get something like:

I roast a whole chicken in a pot. I roast a whole chicken in a pot with a lid on it. I roast a whole chicken in a pot with a lid on it.

If you're not sure what a pot is, ask your butcher. Sometimes, when I roast meat, I roast it in a pot, which is more like a refrigerator or freezer, and I roast it in a pot that is a small freezer, or a big freezer, or larger freezer.

Why do I have to be afraid of chickens?

Because they're so adorable.

— GPT-2

Sometimes it answers the question. Sometimes it continues your question with three more questions. Sometimes it drifts into a Reddit thread it had hallucinated. While it could make for an interesting conversation, it's not something that can be deployed.

Luckily, we know exactly how this problem was solved: post-training, specifically, supervised instruction tuning, RL from human feedback, and RL with verifiable rewards, which turned “impressive demos” into “things you can actually use.”

The more important part is that the field converged on a shared for post-training language models:

• Start from a strong pretrained model.
• Define the environments and reward, i.e. verifiable or learned preference models.
• Run RL optimization with a specific family of algorithms, anchored to the reference model.
• Watch for and address known pathologies such as reward hacking.

With this concrete recipe, anyone who wants to fine-tune the latest LLM on a downstream task has a concrete path to follow, with documented failure modes and sane defaults at every step. That is what makes LLM post-training tractable at scale.

Robotics is sitting almost exactly where language modeling was: The pretraining has scaled beautifully. We already have very good pretrained policies — vision-language-action models (VLAs) and world-action models (WAMs) trained on enormous piles of data — and they execute complex behaviors. What is missing is the other half: the model learning from its own experience, and for that, we need a recipe for post-training. And for robotics, it needs to be even more reliable than language models. A bad script of code generated by an LLM gets caught by a human reviewer before it goes into production. A bad robot action doesn't wait for human review and just happens.

So why hasn't this happened yet?

A lot of recent advancement in robotics has been driven by imitation learning, which inherited nearly for free everything that made supervised learning so clean and easy to use following development of techniques such as action chunking and good teleoperation interfaces. The loss doesn’t explode for seemingly mysterious reasons; you know the few hyperparameters that are important for performance; and loss curves correlate with performance.

RL shows incredible promise in training frontier robotics models, but it is not yet a recipe. It's a craft.

Getting it to work takes a lot of intuition, just like how a good chef seasons by feel. What that means in practice is that getting RL post-training to work reliably is limited to a small number of people. This was once true for LLMs, until the field converged on a shared recipe.

For robotics to be deployable at scale, we think it needs what language modeling has: a universal post-training recipe.

The remainder of this post is about closing that gap, what RL for frontier robotics models actually looks like today, why it is different from RL for language models, and what it will take to turn the craft into a recipe.

Two things a recipe needs

We think closing the gap from craft to recipe takes two things.

The first is an algorithm built specifically for fine-tuning frontier robotics models, one that stays stable when applied to models with billions of parameters, and that learns from a small enough amount of experience to be practical on real hardware, where every attempt costs valuable time on a real robot.

The second is easier to overlook, but just as important: a set of standard practices surrounding the algorithm. A default way to define what counts as success. A default way to reset the scene between attempts, so the robot can try again. A default way for a person to give feedback, and to turn that feedback into learning. The algorithm plus the protocol around it defines a set of concrete defaults for bringing frontier robotics post-training to scale.

Why is robotics a different RL problem than LLMs?

But haven’t we already figured out large-scale RL? RL post-training has driven most of the recent gains in language models, at real scale and in production. Why would RL for robotics be any different?

To answer that, it helps to step back and look at where deep RL first found its footing. One of the first deep reinforcement learning systems to gain broad traction was AlphaGo, which beat Lee Sedol at Go in 2016. AlphaGo learned a policy to predict the optimal action in a given position from a massive number of repeated self-play games. This is quite similar to RL for language models, which uses a massive number of parallel text generations to see which responses get a good reward. In both cases, actions are cheap to generate and cheap to verify, which is an essential ingredient for the type of reinforcement learning that already works at scale.Learning through massively parallel simulationevery square is a full game of Go, sampled from the current policy and scored the instant it ends288 games in parallel0 games scoredpolicy v0black wins white wins On-policy RL where samples are cheap: thousands of games are generated in parallel, each one is verified by the rules of the game for free, and after every update the old games are thrown away and a fresh batch is played by the new policy.

The type of RL that has already been demonstrated at scale is on-policy policy gradient RL, which learns by hill-climbing directly on reward, and requires freshly sampled data for every update. That's fine for Go and for LLMs. Robotics is a different story because data is much more expensive to generate. A robot learning to fold laundry in the real world cannot try a thousand different actions at each low-level control step, especially when a single task chains together thousands of steps or more, and while people have explored simulations, modeling real world objects accurately in simulation can often be even harder than learning the task itself. As a result, there is a significant focus on the ability to leverage past data, for example data from many update steps ago of the policy being learned.Even a simple task can take 500 decisionspicking up a glass looks like one action, but the policy has to output 500 small commands in a row, and the only reward comes at the end500 control steps7-D action every 20 msreward 0positive command negative command A single episode of a simple manipulation task. The policy outputs a fresh action at every control step (orange dots on the gripper's path), but success is only observed at step 500. To learn from that one signal, the agent has to work out which of the 500 preceding actions were responsible.

On top of that, LLM RL and robotics RL are solving different-shaped problems. RL for language models has typically been framed as generating , with a reward assigned to that response. Robotics involves generating hundreds to thousands of low-level actions or more for a single task. The reward the policy needs to learn from usually arrives only at the very end indicating task success. On top of that, environments for tasks like language or Go are deterministic, and if you generate an action, that action gets played. In robotics, the environment itself can be stochastic, which means even sending the exact same command to the robot twice can produce slightly . This means the agent has to assign credit to actions taken thousands of steps earlier, across variations in the environment that compound over the horizon.

The combination of expensive samples and long-horizon structure pushes the field toward value-based RL: methods that learn scoring models (value functions) to assign credit across long horizons. And value-based RL at meaningful scale has been validated far less than the large-scale RL post-training regimes of LLMs.

Why can we not use existing value-based RL approaches?

Around the same era that PPO, the predominant method for LLM post-training, was developed, a parallel line of value-based methods, for example DDPG, TD3, and SAC, emerged for robotics control, and these have been shown to work on real-world tasks. But applying them directly to frontier robotics models runs into two problems.

The first is how these models represent their decisions. The older algorithms assume the policy draws actions from a Normal distribution: one average action plus some noise around it. That's fine when the right behavior really is one thing with a bit of noise. But modern robotics models are built for situations where there are many types of correct action possibilities. For example, imagine the task of grasping a cup, one can grasp it by the handle or by the base, and both would be correct. To account for this, frontier robotics models use richer machinery like diffusion for action generation (the same method behind image generation that try to generate samples from noise) instead of a simple bell curve. The older RL methods simply weren't built to work for these models.

The second is stability. Deep RL has a well-known tendency for instability, and this is especially the case as models get bigger. Value-based RL methods make this worse in a specific way: instead of learning purely from real outcomes, they partly learn by predicting their own future predictions, and reusing those predictions as if they were ground truth. Small errors in that process compound over time. This stability gets worse when you scale to larger models which correlates with more competent policies, such as billion-parameter VLA policies.

have tried to use value-based learning with frontier robotics models such as VLAs. One pushes the learning signal back through the model's entire generation process, but tends to be unstable as the signal has to travel through a long chain of denoising steps. The second leaves the model alone entirely, for example generating a handful of candidate actions and executing whichever one the value function scores highest. While safe, they leave performance on the table, since the model itself never actually learns which actions are good and which actions are bad.

A third option is to learn noise that gets fed into the model that determines downstream action generation. Instead of touching the model's weights, you use RL to search for better noise to feed it. This is appealing because it's low-risk. No matter the noise selected, the result is still an action the model already knows how to produce, so it is harder to get something unreasonable. It also works reasonably well in practice, because searching over noise is a much easier problem than searching over actions directly. But that comes at a cost. You can only ever recombine behaviors the model already has some version of. If the skill a robot needs isn't already somewhere in what the base model learned during pretraining, no amount of noise-searching will produce the action, and both how good the robot ultimately gets, and how quickly it improves, are capped by whatever the original model already knew.

To go beyond what the model already knows, the learning process needs to update its weights directly in a stable way, so it can use the model's full capacity to represent complex, varied behavior.

What might an algorithm for robotics RL look like?

For an algorithm to post-train frontier robotics models well, the key is to have a stable way to improve large policies that rely on modern generative techniques, such as diffusion, using estimates of how good an action is. is a system we built based on what we imagined that algorithm might look like, and we tested it on real robots, doing real tasks: routing a string of holiday lights through hooks and plugging it in to light it up, striking a pool ball into a pocket, inserting a flower into the neck of a wine bottle, flipping an egg. Going in, frontier pretrained models could not reliably complete these tasks.

EXPO(-FT) works by learning to repeatedly improve actions from the frontier model using reinforcement learning with small edits from a lightweight policy, and then absorbing that into the frontier model itself. To get the best possible action during execution, the robot generates a handful of candidate actions from the frontier model, produces a higher-value edited version of each, and uses the value function to estimate how well each candidate will turn out and pick the best one. Because the edits are deliberately kept small, a nudge can't send the robot to execute something dangerous and all the volatility of RL stays confined to the small model. And because the pretrained model keeps training on the actions that were selected and worked, improvements discovered by the edits get absorbed into the base model itself. This means the model will make better proposals next time, letting the large pretrained model bring its full capacity to bear on learning new behavior.The edit policy proposes small, bounded adjustments to the base model's actions, nudging each one from a low-value region toward the higher-value peak of its own mode (orange dots), without ever straying far from what the base model already knows how to do.

Throughout online learning, a human supervises the robot and can intervene whenever it starts to go wrong. Those interventions are fed back into training to improve the policy further.

Across six complex manipulation tasks, such as routing string lights, sinking a pool shot, and inserting a flower into a bottle, EXPO-FT reaches 30/30 success using an average of only 19 minutes of online interaction.Ball Balancing · EXPO-FTBall Balancing · π0.5 Supervised FinetunedString Light Routing · EXPO-FTString Light Routing · π0.5 Supervised FinetunedAverage success rate across tasks: EXPO-FT reaches 30/30, compared with SFT, HG-DAgger, DSRL, and HIL-SERL.

Practically, this means a frontier model can be deployed on tasks like these and then post-trained to high reliability in a matter of tens of minutes.

What does this not solve? EXPO-FT is our attempt at the first of the two things we said a recipe needs, and it can inform what the rest of the work needed might look like.

First, a human is in the loop to a large extent. Someone defines success, resets the scene, and intervenes when the robot goes wrong. That doesn't mean it cannot scale. Something like Waymo, for example, can run using remote operators with one person who can oversee many vehicles at once. The robot can perform learning autonomously, but going from one robot and one operator to a large fleet depends on how much human attention each additional robot demands. This is part of the second half of the recipe, the protocol, and there are a lot of questions that remain.

Second, longer horizons strain the value function. EXPO-FT is safe because it never moves far from the base model in any single step, but that safety property is only as good as the value estimates behind it. Longer horizons make credit assignments harder and push training times up. Assigning credit across thousands of steps or more with reward arriving only at the end is the goal, and we tested the shorter-horizoned end of it.

Finally, the computational cost can be high. Nineteen minutes of robot interaction is not nineteen minutes of post-training. Gradient updates on a model this large dominate wall-clock time, and closing that gap matters for post-training to become a routine.

We think EXPO-FT shows that stable RL post-training on a frontier robotics policy is achievable. Turning that into something a team can pick up and expect to work on their own robot and their own task requires standardized training protocols, as in the case for LLMs.

Standard training protocols

Even with the best algorithm, post-training only becomes a complete recipe with a standard set of protocols around it: how a task counts as successful, how resets are done, how humans provide input, how to tune hyperparameters, and how to initialize the task and policy.

Reward specification. Reinforcement learning works by optimizing rewards, and different rewards produce vastly different outcomes even for the same task. In LLM RL, reinforcement learning from verifiable rewards (RLVR) gave the field a default answer: check the answer, check the tests. Robotics has no equivalent. Today, success detectors are either hand-built per task or replaced by a human watching each trajectory and calling it, neither of which scales. Learned success classifiers and reward models are all plausible directions to explore.

Resets. Between episodes, the robot needs to be reset, and today that is usually a person's job. Open questions remain both on what states to reset to and on how to get there without a human, for example via a learned reset policy or reversible task design. Perhaps for real deployment, a reset is not even needed and instead moving onto the next task without undoing the previous is the better alternative. Determining which of those choices are the best matters for a universal recipe.

Human in the loop. Human interventions have proven to be a useful tool for improving data efficiency. How much to provide, when to provide it, and how that signal gets used during training are open questions with direct implications for how any of this scales past one robot and one operator.

Hyperparameter tuning. Value-based RL is known to be sensitive to hyperparameters. Learning rate, the update-to-data ratio, when to stop training, the task horizon, and the control frequency at which the policy acts all shape how stable and sample-efficient fine-tuning turns out to be, and none of them has a well-understood default in this setting the way they do for supervised learning.

Initialization. How the task and policy are initialized can play a large part in how performance turns out. How much data to initialize from, what that initial dataset should contain, and how heavily to weigh it against newly collected online experience are all still open questions that already have answers in language modeling, but not yet in robotics.

None of these has a default answer yet, and the right answer may also depend on the algorithm. We’ve created a set of reference tuning tips as a step towards this direction for EXPO-FT, but that does not completely address these problems. We think they are among the most consequential open problems in the field. They are also the kind of problem that gets solved through community effort, by converging on shared answers.

Towards universal post-training

Language model post-training became scalable because the field settled on defaults concrete enough to follow and be expected to work. Robotics is arriving at the same moment: the models are large, general, and pretrained, and people are trying to make them deployable; and robotics deployment needs to be even more reliable than language models, because robots acting autonomously in the world can't be reviewed the way we review writing from LLMs.

This is exactly when a standardized recipe matters most. Pretraining gave robotics models that know how to do almost anything at once. Post-training is how they learn to do things reliably every time. We believe it is what will bring frontier robotics to where LLMs are today and beyond, and converging on a set of industry defaults, a universal post-training recipe, is the most important part of bringing us to that point.

Acknowledgements

Thanks to Anikait Singh, Aneesh Muppidi, Dion Dong, and Jules Qiu for helpful discussions and feedback on this post.

Perry Dong

PhD student in Computer Science at Stanford University, working on reinforcement learning. More about PerryMore posts