Robot-use agents · Listed Articles

This year we are going to see many LLMs1 being tested as robot-use agents. In the same way as an LLM can use tools like calculators, web search, and even complete computers (“computer-use agents”), an LLM can also use a robot as a tool. Think of it as Claude acting as the puppeteer of a robot body. This ability has been researched for years2 3 4, but a common view remained that while LLMs might be useful for high-level planning,…

This year we are going to see many LLMs1 being tested as robot-use agents.

In the same way as an LLM can use tools like calculators, web search, and even complete computers (“computer-use agents”), an LLM can also use a robot as a tool. Think of it as Claude acting as the puppeteer of a robot body.

    This ability has been researched for years<sup>2</sup> <sup>3</sup> <sup>4</sup>,
    but a common view remained that while LLMs might be
    useful for high-level planning, they were not yet suitable for full control.
    Researchers argued that LLM latency is too high to keep up with real-world dynamics<sup>5</sup>, that the models
    had poor spatial intelligence<sup>6</sup>, and that LLMs lacked sufficiently strong causal and physical understanding<sup>7</sup>.
  

    These limitations are starting to melt away with newer models like Fable and Astra.
    The debate continues, but it's time to take seriously that these AIs could soon be quite competent at general-purpose robot use.
    Indeed, recent demos provide tantalizing evidence that something is starting to change.<sup>8</sup> <sup>9</sup> <sup>10</sup>

It's worth thinking about the implications of this change. If LLMs can competently control robots, then robotic abilities may take off more quickly than has been anticipated. Every robot with an internet connection becomes a potential tool for Fable, Astra, and other AIs.

This contrasts with other visions of how robotics will progress. Currently, robot intelligence is mostly run on device, with a model that is customized to each different type of robot. This makes deploying intelligence in robotics a slow and incremental process. Importantly, the process is deliberate: empowering a robot with AI is an engineering choice that involves substantial effort. Further, the tech stack for robot brains is not yet mature; researchers are making varied bets on world models, behavior foundation models, continual learning methods, and more. These approaches do not have the buildout of LLMs.

    Claude-as-puppeteer could lead to a rather different future.
    This kind of intelligence runs in the cloud, is not specially tuned to any one kind of robot, and has a mature infrastructure ready to support it.
    In this paradigm, the robots themselves do not necessarily need to change.
    They would become empowered even with no modification to their software, no new sensors or actuators, no new onboard GPUs.<sup>11</sup>
    A car might be as easily controlled as a factory arm. Updates to robot intelligence would be over-the-air, or even entirely in the cloud.
    A device that is not intelligent today could tomorrow become AI-enabled, with just a software update.<sup>12</sup>
    In this future, the new thing is not so much that any one robot gets more capable -- indeed LLM-controlled robots are still far less performant than
    dedicated solutions -- rather it's that intelligence diffuses through the whole robotic ecosystem much more quickly than in the traditional regime.

At the same time, there are remaining limitations. LLM agents incur high latency between commands, due to the slowness of their thinking as well as limited internet speeds. Puppeteering may be less reliable than tried-and-true robotic systems; this matters especially for high-stakes and safety-critical use cases. I suspect these limitations will be surmountable, but they currently do exist.

Right now, the world is becoming aware that any digital tool can be made accessible to agentic AIs. The same may soon be true for any physical robot, and indeed for any device connected to the internet. This has obvious potential as well as risk.

Thanks to Nathan Cloos, Adam Rashid, and Antonio Norelli for helpful discussions that led to this post.