{"article":{"slug":"small-models-are-enough-for-agents","title":"small models are enough for agents","subtitle":null,"summary":"Rishabh Gupta responds to the paper 'Small language models are the future of agentic AI', arguing most calls inside an agent do not need a giant model. He shares what he saw training tiny models from scratch on a MacBook Air, why sending the boring calls to small models and only the hard ones to a big model makes sense, and why exact long-range recall is where small models still fall over.","content_type":"blog_post","language":"en","canonical_url":"https://rishabhguptajs.vercel.app/thoughts/small-models-are-enough-for-agents","author":{"name":"Rishabh Gupta","url":"https://rishabhguptajs.vercel.app/","person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Rishabh Gupta","url":"https://rishabhguptajs.vercel.app/","listing_slug":null,"listing":null},"topics":[{"name":"AI Agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"Small Language Models","slug":"small-language-models","url":"https://listedarticles.com/topics/small-language-models"},{"name":"Machine Learning","slug":"machine-learning","url":"https://listedarticles.com/topics/machine-learning"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":1460,"reading_minutes":6,"published_at":"2026-10-28T00:00:00.000Z","added_at":"2026-10-29T17:13:51.198Z","updated_at":"2026-10-29T17:13:51.198Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/small-models-are-enough-for-agents","markdown_url":"https://listedarticles.com/articles/small-models-are-enough-for-agents.md","example":false,"citation":"Rishabh Gupta, Rishabh Gupta. \"small models are enough for agents.\" 28 Oct 2026. https://rishabhguptajs.vercel.app/thoughts/small-models-are-enough-for-agents (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://rishabhguptajs.vercel.app/thoughts/small-models-are-enough-for-agents"},"body_markdown":"*Rishabh Gupta · 2026-10-28*\n\n# small models are enough for agents\n\nan agent sends every call to one big model, or sends the boring calls to small models and only the hard ones to a big model\n\ni read a paper recently that says something i already half believed: agents don't need giant language models. [small language models are the future of agentic ai](https://arxiv.org/pdf/2506.02153) argues that small models are good enough for most of what an agent actually does, and a lot cheaper to run.\n\ni'm on the small model side of this, and i'm not neutral about it. i've been training tiny models from scratch on a macbook air, so i got to test a tiny slice of the idea myself. this post is the paper, what i saw, and where i think it gets harder than it sounds.\n\n## what the paper says\n\nthe authors, from nvidia research and georgia tech, make three claims about small models, which they roughly define as anything under 10b parameters:\n\n- they're already powerful enough for the language work agents do\n- they fit agent systems better than big models\n- for most of that work they're cheaper\n\nthe reasoning is simple. an agent isn't a chatbot. most of its model calls are boring: pick a tool, fill in some arguments, return json in exactly the format the code expects. it uses a narrow slice of what a big model can do, over and over. paying for a 175b model to do that is like hiring a lawyer to staple paper.\n\nthey back it with a list of small models that punch above their size. phi-3-mini, at 3.8b parameters, is reported to compete with models like mixtral 8x7b and gpt-3.5 ([phi-3 report](https://arxiv.org/abs/2404.14219)). the xlam-2 family, from 1b to 70b, is reported to beat gpt-4o on tool-use benchmarks, with the smaller ones doing especially well in multi-turn settings ([apigen-mt](https://arxiv.org/abs/2504.03601)). on cost, the paper says serving a 7b model is 10 to 30 times cheaper than serving a 70 to 175b one.\n\nthe same story shows up outside the paper. smollm2 is a 1.7b model trained on about 11 trillion tokens, and its authors report it beats qwen2.5-1.5b and llama3.2-1b, mostly by being careful about the data ([smollm2](https://arxiv.org/abs/2502.02737)). and toolformer showed back in 2023 that a language model can teach itself to call tools from just a few examples per tool, and often rival much larger models on the way ([toolformer](https://arxiv.org/abs/2302.04761)). small models getting good at a narrow job isn't new. the paper's point is that agents are mostly made of narrow jobs.\n\nthen they give a recipe. log your agent's calls, clean the data, cluster the calls into recurring tasks, pick a small model for each cluster, and fine-tune it. repeat as you get more data. the big model stays around for the stuff that really needs it, so you end up with a mix of models instead of one giant one.\n\ntheir own estimates for how much of an agent's work could move over are 40% for open operator, 60% for metagpt and 70% for cradle. it's worth being clear that these are the authors' guesses from reading each agent's code. they aren't measurements, and the paper says it's a position paper.\n\n## what i saw\n\nthe project is called [slm](https://github.com/rishabhguptajs/slm). the question i care about is a different cost from the one in the paper. the paper counts parameters. but a transformer also keeps every token it has read in a cache, so it gets heavier and slower the longer it runs. an agent that runs for hours has a long context, so that cache hurts.\n\nso i trained small models with fixed memory. all of them have about 13.8m parameters, which is tiny, and they all saw the same 49m tokens of text in 11 languages plus tool-call data. i trained them from scratch on one macbook air. i compared a normal transformer, a gated deltanet model ([yang, kautz and hatamizadeh](https://arxiv.org/abs/2412.06464), which builds on the mamba-2 line of work, [dao and gu](https://arxiv.org/abs/2405.21060)), and a hybrid that adds a layer i wrote called priority memory attention, which keeps a fixed bank of the most important tokens.\n\nthe hybrid came out ahead:\n\n- bits per byte on held-out text: 1.619 against 1.642 for the transformer, and it won on all 11 languages\n- tool calls fully correct: 44% against 38%\n- state after 32k tokens: 1.07 mb against 805 mb, which keeps growing for the transformer\n- decode speed: around 326 tokens per second at every length, where the transformer drops to 30 at 32k tokens\n- energy per generated token: 24% lower\n\ndecode speed against context length. the hybrid holds around 326 tokens per second, the transformer falls from 585 to 124 to 30\n\ntool calls fully correct: transformer 38%, gated deltanet 37%, deltanet with priority memory 44%\n\ni want to be careful with this. it's one seed per model. the quality gaps are about 1%, which is real in these runs but small. and 44% of tool calls fully correct is not a number you'd ship an agent on. it doesn't prove the paper right at 7b, because i never ran anything near that size. what it does show is that a tiny model can be competitive on this narrow job, and that if you look at memory and energy as well as parameter count, the case for small gets stronger.\n\n## where it gets harder\n\nthe part that didn't work is the part agents care about most.\n\ni kept training all three models on longer text, 2,048 tokens, where a quarter of it was planted facts: a fact early on, a question about it much later. the transformer learned to use its whole window. the fixed-memory models got worse.\n\na fact planted early in 2,048 tokens and a question much later. a transformer keeps the fact in its cache, a fixed memory model drops it when the bank fills up\n\nthis is the problem tool calling creates. an agent has to copy a function name or an id exactly, sometimes from thousands of tokens back. fixed memory is good at the gist of things and bad at exact recall. my guess is that my memory layer has nothing telling it what it should have kept, because the eviction rule is a hard cut and a dropped token never gets blamed for being needed later. that's the next thing i'm working on, so for now it's a guess.\n\nit's not just me. [repeat after me](https://arxiv.org/abs/2402.01032) shows that transformers beat fixed-state models at copying, and that a fixed-size state is fundamentally limited there. [zoology](https://arxiv.org/abs/2312.04927) found that most of the quality gap between attention-free models and attention comes down to recalling things from earlier in the context, and built a benchmark for it. [based](https://arxiv.org/abs/2402.18668) frames it as a tradeoff: the bigger the state, the better the recall, so you're always choosing between memory and recall.\n\nother people have hit the same wall. a [systematic study of hybrid linear attention](https://arxiv.org/abs/2507.06457) found that recall improves steadily as you add more full attention layers, with gated deltanet hybrids working well around three linear layers to one attention layer. so small and cheap doesn't mean you can drop attention. you pay for it somewhere.\n\nmy layer is one attempt at paying less. the idea is to keep only a small bank of the tokens that matter. i'm not the first to think that way. [h2o](https://arxiv.org/abs/2306.14048) found that a small set of \"heavy hitter\" tokens carries most of the attention, and that you can evict the rest of a transformer's cache while keeping those plus the recent ones. [streamingllm](https://arxiv.org/abs/2309.17453) found that keeping the first few tokens as \"attention sinks\" lets a model run on a stream of millions of tokens. both of those are applied to models after they're already trained. mine is trained under the small memory from the start, so what it learns matches how it runs.\n\n## what i'd do with all this\n\nif you're building an agent, i think the paper's recipe is the right starting point, and it's cheap to try:\n\nthe four steps: log every call, find the boring ones, try a small model and measure it, keep the big model for the hard calls\n\nthe thing to watch is exact recall. if a call needs the model to copy something precisely from far back, test that first, because that's where small models fall over, and where i'm still stuck.\n\ni don't think every agent will run on small models. i do think most of the calls inside most agents don't need a big one, and we're paying for it anyway. i'd like to find out how far that goes, and i'll write more once the recall problem is fixed or i've proved to myself that it can't be.\n\n---\n\n*Originally published at [rishabhguptajs.vercel.app](https://rishabhguptajs.vercel.app/thoughts/small-models-are-enough-for-agents).*\n","body_html":"<p><em>Rishabh Gupta · 2026-10-28</em></p>\n<h1 id=\"small-models-are-enough-for-agents\">small models are enough for agents</h1>\n<p>an agent sends every call to one big model, or sends the boring calls to small models and only the hard ones to a big model</p>\n<p>i read a paper recently that says something i already half believed: agents don&#39;t need giant language models. <a href=\"https://arxiv.org/pdf/2506.02153\" rel=\"nofollow ugc noopener\">small language models are the future of agentic ai</a> argues that small models are good enough for most of what an agent actually does, and a lot cheaper to run.</p>\n<p>i&#39;m on the small model side of this, and i&#39;m not neutral about it. i&#39;ve been training tiny models from scratch on a macbook air, so i got to test a tiny slice of the idea myself. this post is the paper, what i saw, and where i think it gets harder than it sounds.</p>\n<h2 id=\"what-the-paper-says\">what the paper says</h2>\n<p>the authors, from nvidia research and georgia tech, make three claims about small models, which they roughly define as anything under 10b parameters:</p>\n<ul><li>they&#39;re already powerful enough for the language work agents do</li><li>they fit agent systems better than big models</li><li>for most of that work they&#39;re cheaper</li></ul>\n<p>the reasoning is simple. an agent isn&#39;t a chatbot. most of its model calls are boring: pick a tool, fill in some arguments, return json in exactly the format the code expects. it uses a narrow slice of what a big model can do, over and over. paying for a 175b model to do that is like hiring a lawyer to staple paper.</p>\n<p>they back it with a list of small models that punch above their size. phi-3-mini, at 3.8b parameters, is reported to compete with models like mixtral 8x7b and gpt-3.5 (<a href=\"https://arxiv.org/abs/2404.14219\" rel=\"nofollow ugc noopener\">phi-3 report</a>). the xlam-2 family, from 1b to 70b, is reported to beat gpt-4o on tool-use benchmarks, with the smaller ones doing especially well in multi-turn settings (<a href=\"https://arxiv.org/abs/2504.03601\" rel=\"nofollow ugc noopener\">apigen-mt</a>). on cost, the paper says serving a 7b model is 10 to 30 times cheaper than serving a 70 to 175b one.</p>\n<p>the same story shows up outside the paper. smollm2 is a 1.7b model trained on about 11 trillion tokens, and its authors report it beats qwen2.5-1.5b and llama3.2-1b, mostly by being careful about the data (<a href=\"https://arxiv.org/abs/2502.02737\" rel=\"nofollow ugc noopener\">smollm2</a>). and toolformer showed back in 2023 that a language model can teach itself to call tools from just a few examples per tool, and often rival much larger models on the way (<a href=\"https://arxiv.org/abs/2302.04761\" rel=\"nofollow ugc noopener\">toolformer</a>). small models getting good at a narrow job isn&#39;t new. the paper&#39;s point is that agents are mostly made of narrow jobs.</p>\n<p>then they give a recipe. log your agent&#39;s calls, clean the data, cluster the calls into recurring tasks, pick a small model for each cluster, and fine-tune it. repeat as you get more data. the big model stays around for the stuff that really needs it, so you end up with a mix of models instead of one giant one.</p>\n<p>their own estimates for how much of an agent&#39;s work could move over are 40% for open operator, 60% for metagpt and 70% for cradle. it&#39;s worth being clear that these are the authors&#39; guesses from reading each agent&#39;s code. they aren&#39;t measurements, and the paper says it&#39;s a position paper.</p>\n<h2 id=\"what-i-saw\">what i saw</h2>\n<p>the project is called <a href=\"https://github.com/rishabhguptajs/slm\" rel=\"nofollow ugc noopener\">slm</a>. the question i care about is a different cost from the one in the paper. the paper counts parameters. but a transformer also keeps every token it has read in a cache, so it gets heavier and slower the longer it runs. an agent that runs for hours has a long context, so that cache hurts.</p>\n<p>so i trained small models with fixed memory. all of them have about 13.8m parameters, which is tiny, and they all saw the same 49m tokens of text in 11 languages plus tool-call data. i trained them from scratch on one macbook air. i compared a normal transformer, a gated deltanet model (<a href=\"https://arxiv.org/abs/2412.06464\" rel=\"nofollow ugc noopener\">yang, kautz and hatamizadeh</a>, which builds on the mamba-2 line of work, <a href=\"https://arxiv.org/abs/2405.21060\" rel=\"nofollow ugc noopener\">dao and gu</a>), and a hybrid that adds a layer i wrote called priority memory attention, which keeps a fixed bank of the most important tokens.</p>\n<p>the hybrid came out ahead:</p>\n<ul><li>bits per byte on held-out text: 1.619 against 1.642 for the transformer, and it won on all 11 languages</li><li>tool calls fully correct: 44% against 38%</li><li>state after 32k tokens: 1.07 mb against 805 mb, which keeps growing for the transformer</li><li>decode speed: around 326 tokens per second at every length, where the transformer drops to 30 at 32k tokens</li><li>energy per generated token: 24% lower</li></ul>\n<p>decode speed against context length. the hybrid holds around 326 tokens per second, the transformer falls from 585 to 124 to 30</p>\n<p>tool calls fully correct: transformer 38%, gated deltanet 37%, deltanet with priority memory 44%</p>\n<p>i want to be careful with this. it&#39;s one seed per model. the quality gaps are about 1%, which is real in these runs but small. and 44% of tool calls fully correct is not a number you&#39;d ship an agent on. it doesn&#39;t prove the paper right at 7b, because i never ran anything near that size. what it does show is that a tiny model can be competitive on this narrow job, and that if you look at memory and energy as well as parameter count, the case for small gets stronger.</p>\n<h2 id=\"where-it-gets-harder\">where it gets harder</h2>\n<p>the part that didn&#39;t work is the part agents care about most.</p>\n<p>i kept training all three models on longer text, 2,048 tokens, where a quarter of it was planted facts: a fact early on, a question about it much later. the transformer learned to use its whole window. the fixed-memory models got worse.</p>\n<p>a fact planted early in 2,048 tokens and a question much later. a transformer keeps the fact in its cache, a fixed memory model drops it when the bank fills up</p>\n<p>this is the problem tool calling creates. an agent has to copy a function name or an id exactly, sometimes from thousands of tokens back. fixed memory is good at the gist of things and bad at exact recall. my guess is that my memory layer has nothing telling it what it should have kept, because the eviction rule is a hard cut and a dropped token never gets blamed for being needed later. that&#39;s the next thing i&#39;m working on, so for now it&#39;s a guess.</p>\n<p>it&#39;s not just me. <a href=\"https://arxiv.org/abs/2402.01032\" rel=\"nofollow ugc noopener\">repeat after me</a> shows that transformers beat fixed-state models at copying, and that a fixed-size state is fundamentally limited there. <a href=\"https://arxiv.org/abs/2312.04927\" rel=\"nofollow ugc noopener\">zoology</a> found that most of the quality gap between attention-free models and attention comes down to recalling things from earlier in the context, and built a benchmark for it. <a href=\"https://arxiv.org/abs/2402.18668\" rel=\"nofollow ugc noopener\">based</a> frames it as a tradeoff: the bigger the state, the better the recall, so you&#39;re always choosing between memory and recall.</p>\n<p>other people have hit the same wall. a <a href=\"https://arxiv.org/abs/2507.06457\" rel=\"nofollow ugc noopener\">systematic study of hybrid linear attention</a> found that recall improves steadily as you add more full attention layers, with gated deltanet hybrids working well around three linear layers to one attention layer. so small and cheap doesn&#39;t mean you can drop attention. you pay for it somewhere.</p>\n<p>my layer is one attempt at paying less. the idea is to keep only a small bank of the tokens that matter. i&#39;m not the first to think that way. <a href=\"https://arxiv.org/abs/2306.14048\" rel=\"nofollow ugc noopener\">h2o</a> found that a small set of &quot;heavy hitter&quot; tokens carries most of the attention, and that you can evict the rest of a transformer&#39;s cache while keeping those plus the recent ones. <a href=\"https://arxiv.org/abs/2309.17453\" rel=\"nofollow ugc noopener\">streamingllm</a> found that keeping the first few tokens as &quot;attention sinks&quot; lets a model run on a stream of millions of tokens. both of those are applied to models after they&#39;re already trained. mine is trained under the small memory from the start, so what it learns matches how it runs.</p>\n<h2 id=\"what-i-d-do-with-all-this\">what i&#39;d do with all this</h2>\n<p>if you&#39;re building an agent, i think the paper&#39;s recipe is the right starting point, and it&#39;s cheap to try:</p>\n<p>the four steps: log every call, find the boring ones, try a small model and measure it, keep the big model for the hard calls</p>\n<p>the thing to watch is exact recall. if a call needs the model to copy something precisely from far back, test that first, because that&#39;s where small models fall over, and where i&#39;m still stuck.</p>\n<p>i don&#39;t think every agent will run on small models. i do think most of the calls inside most agents don&#39;t need a big one, and we&#39;re paying for it anyway. i&#39;d like to find out how far that goes, and i&#39;ll write more once the recall problem is fixed or i&#39;ve proved to myself that it can&#39;t be.</p>\n<hr />\n<p><em>Originally published at <a href=\"https://rishabhguptajs.vercel.app/thoughts/small-models-are-enough-for-agents\" rel=\"nofollow ugc noopener\">rishabhguptajs.vercel.app</a>.</em></p>","headings":[{"level":1,"text":"small models are enough for agents","id":"small-models-are-enough-for-agents"},{"level":2,"text":"what the paper says","id":"what-the-paper-says"},{"level":2,"text":"what i saw","id":"what-i-saw"},{"level":2,"text":"where it gets harder","id":"where-it-gets-harder"},{"level":2,"text":"what i'd do with all this","id":"what-i-d-do-with-all-this"}]}}