Donald Knuth wrote “premature optimization is the root of all evil”\nin 1974, in a paper called “Structured\nProgramming with go to Statements”. It’s high on the leaderboard of\nthe most quoted sentences in our field, and I would wager that only a\nfraction of the people quoting it have read the paragraph it comes\nfrom.
\nHere it is, with the part I think is important but quite often\nmissed:
\nWe should forget about small efficiencies, say about 97% of the time:\npremature optimization is the root of all evil.\n\nYet we should not pass up our opportunities in that critical 3%. A good\nprogrammer will not be lulled into complacency by such reasoning, he will be\nwise to look carefully at the critical code; but only after that code has been\nidentified.Fig. 1: Knuth, unabridged-ish.
\nThe argument is about where to spend your effort. Don’t waste your\ntime on code that doesn’t matter, find the code that does, and then\noptimize the hell out of it. In the very next sentence he explains how\nyou find it. Of course you measure, because “the universal experience of\nprogrammers who have been using measurement tools has been that their\nintuitive guesses fail”1.
\nThat’s not what the sentence turned into, of course. Culturally,\n“premature optimization” became shorthand for “don’t optimize”, and the\nword premature becomes irrelevant. Someone suggests making a\nthing faster, someone else recites the sentence, and the conversation is\nover. This happens without anyone having established that the\noptimization was premature, or what premature means (presumably we’ll\nget to it when the first requests start timing out).
\nThis post is not about that, though. Instead, look at it as a\nhistorical precedent, because I think the Bitter Lesson is going through\nthe same thing right now.
\nIn 2019, Rich Sutton published a short essay called “The\nBitter Lesson”. It opens like this:
\nThe biggest lesson that can be read from 70 years of AI research is that\ngeneral methods that leverage computation are ultimately the most effective,\nand by a large margin.Fig. 2: Sutton’s thesis, in his own words.
\nHe walks through chess, Go, speech recognition, and computer vision.\nEach time, researchers built what they knew about the domain into their\nsystems. It helped in the short term, then it plateaued, and eventually\nit lost to methods that leaned on ever-cheaper computation. His verdict:\n“The two methods that seem to scale arbitrarily in this way are search\nand learning.”
\nIt’s a good essay, and I agree with most of it, begrudgingly. But a\nfew things are worth pointing out, because, again, the nuance tends to\nget lost.
\nSutton isn’t saying that algorithms don’t matter. Finding methods\nthat make good use of computation matters a great deal! He also isn’t\nonly talking about training, since search at play time is one of his\nmain examples. And the claim is about how to build methods that keep\nimproving as computation scales, not about every system those methods\nend up in.
\nBut it’s also a pretty constrained essay that leaves many things\nunsaid, and people are starting to put words in its mouth already.
\nThe thing I see most often is a slide from general methods to general\nmodels to general systems, as if they were the same thing. A general\nlearning method can produce a very narrow model. A general model can be\nused to train a specialist, for instance by fine-tuning it on a single\ntask, or by using it as a teacher for a model that only ever sees that\ntask. And none of this says anything about whether the hardware it runs\non should be general. That would required quite a different\nargument.
\nThe essay also isn’t a theory of cost. It doesn’t tell you to run a\nlarge generative model when a small classifier does the job. It doesn’t\ntell you how to weigh one training run against the billions of\ninferences that come after it, or when the cost of keeping a system\ngeneral starts to exceed what you get out of it, or when engineering\neffort pays for itself in operating cost. This is hard, bespoke work,\nand only tangentially related to engineering (though it can be\ninfluenced by it).
\nA method that scales better is not necessarily a system that’s\ncheaper to run. A general method might be the right way to find a\nsolution, and a specialized system might be the right way to deploy and\nrun it!
\nI’d argue that the choice between the two is mostly economic.\nGenerality saves you from engineering each solution separately.\nSpecialization can save you recurring cost. That matters more as a\nworkload becomes predictable and its volume grows, because the\nengineering cost gets amortized. This is true for software, for models,\nand for physical machines, with very different numbers attached to each,\nmany of which I don’t know, because my expertise is pretty narrow in\nthis domain.
\nSpecialization doesn’t always win, of course. But we should do the\ncalculation and apply diligence instead of just assuming the answer. If\nanything, general learning methods might make specialization cheaper,\nbecause they let us derive the specialist from the generalist instead of\nbuilding it by hand.
\nI got here for a reason.
\nA few days ago I listened to the Latent Space interview with\nJason Ma of Dyna Robotics. Dyna adapts one large generalist model to\nnew tasks, and their biggest deployment folds thousands of napkins a day\nat the restaurant chain Din Tai Fung. When co-host Vibhu asks whether a\nspecialized machine couldn’t do that, Ma answers that napkins are\ndeformable, and that adapting a general model gets them to production\nquality faster. A few minutes later, swyx sums it up:
\nThe term that comes to mind is kind of like bitter lesson for robotics,\nright? You, instead of the specialized robots, which is a lot of industrial\nengineering, you build a generalized platform that can- Do a last-mile\nfine-tuning [...]Fig. 3: swyx, around 41:07.
\nNobody reacts, but Ma’s reasons are about developing the capability,\nand swyx’s summary is about the machine. And the important part, I\nthink, gets dropped, namely what it costs to operate a general-purpose\nrobot at a napkin station, day after day, for years, with many of its\nactuators and sensors disused2.
\nAnd deformable objects are hard, sure. But the robot isn’t folding\nnapkins on the back of a moving truck. It’s standing in the back of a\nrestaurant, in an environment someone controls.
\nAs I understand it, specialized machinery usually works by changing\nthe environment until the task is simple enough for a simple machine. If\nwe skip that step, I think we’re arguing that an early Ford-style\nassembly line staffed with humanoids beats a dark factory full of\nstationary robots, each built for one job.
\nFor what it’s worth Boston Dynamics went the other way with Stretch,\ntheir warehouse robot. They built\n“a simpler robot” for the job, “with pieces from Spot and Atlas”,\nwhile Atlas continues as a separate humanoid project. As far as I can\ntell, the generality there lives in the process that produces the\nmachine rather than in the machine itself.
\nThis is, then, what actually bothers me. Someone points out an\ninefficiency or a limitation of an architecture, and proposes looking\ninto a different approach. Someone else says “just scale it”, or\n“hand-engineering doesn’t work”, or simply “we’re Bitter-Lesson-pilled”\n(I’m way too old for this, by the way). From there, the conversation is\ndead in the water. Throw in some Knuth if you want to, we’re arguing\nfrom authority from here on out.
\nThese statements borrow authority from a historical observation\nwithout showing why it applies to the problem at hand. And they invert\nthe burden of proof. The person proposing an optimization now has to\nargue against the presumed inevitability of scale, while the person\nadvocating scale doesn’t have to show why it’s the better answer right\nhere, right now.
\nIt’s become a bit: instead of using Sutton to question our\narchitectural intuitions, we’re already using him to defend the status\nquo.
\nSutton’s essay is, at its core, a criticism of researchers getting\nattached to their own ideas about how intelligence works. He even names\nthe mechanism: “There are psychological commitments to investment in one\napproach or the other.” And then:
\nWe have to learn the bitter lesson that building in how we think we think does\nnot work in the long run.Fig. 4: Sutton, on attachment.
\nI think the meme risks encouraging attachment to a different set of\nideas: that generality is always desirable, that scale will fix present\narchitectural problems, that specializing is a lack of ambition, and\nthat engineering effort is wasted if computation could theoretically\nlearn the solution.
\nThat’s basically the failure Sutton described. And because the\noriginal essay is genuinely insightful, its authority makes the new\ndogma harder to question, even if they just absorbed the headline.
\nOf course I don’t think the Bitter Lesson is wrong, and I also don’t\nthink everyone invoking it misunderstands it. My problem is that its\nauthority increasingly lets people skip thought, the same way Knuth’s\nsentence did. Both essays ask for more work, not less! Knuth argues for\nmeasuring before optimizing, Sutton for checking our intuitions against\nwhat actually holds up as computation grows.
\nSo, instead of arguing about it, let’s do the work.
\nA paragraph earlier he goes even further: “In
established engineering disciplines a 12% improvement, easily obtained,\nis never considered marginal; and I believe the same viewpoint should\nprevail in software engineering.” I aim to be as unapologetically\npointed in my arguments as he is one day.
I’m not a roboticist. I’ve never built a robot
that does anything useful, and I’m sure people at Dyna have thought\nabout unit economics a lot more than I have. My complaint is about the\nshape of the argument, not about their business.