![\"a](\"https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/decode_one_token.png\")
It's not because you do less work. It's because you change the kind of work the GPU is doing.
\nBy Joe Barrow | October 8, 2026
\nThere is a common misconception that speculative decoding is fast because you do less work per token.\nThat just “verifying” a draft requires fewer FLOPs than generating the same tokens.
\nThis is untrue.
\nOne of the counterintuitive things about speculative decoding is that you actually do more work!\nThe total number of floating point operations (FLOPs) you execute goes UP for the same sequence.
\nBut then why is it fast? What exactly does an accurate draft model buy you, if it’s not compute efficiency?
\nThe answer lies in the type of work your GPU is doing.
\nSpeculative Decoding is a technique where you use a small draft model to propose a likely draft sequence, and then you use your big model to verify that sequence in parallel (in a single forward pass).
\nTraditional LLM decoding goes one token at a time:
\nBut speculative decoding allows you to look at many tokens at once, and accept the ones that would have been generated by the big model:
\n![\"the](\"https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/speculative_decode.png\")
When a model runs on a GPU, there are different types of work going on.\nThe obvious one is running the matrix multiplies, dot products, or other tensor operations.\nBut an important less obvious one is: loading data from global memory to local memory (and vice versa).
\nWhen the GPU is waiting on data to load for an operation, we call that a memory bound operation.\nIf everything is loaded but we’re waiting on all the tensor operations to finish, we call that a compute bound operation.
\n(For a more in-depth exploration of this, I have a prior post: A Visual Guide to the Roofline Model)
\nWhen an LLM is decoding one token at a time (as in the first figure), it is very memory bound.\nFor every token we have to stream all the active parameters of the model (e.g. 27B parameters for Qwen3.5-27B).\nWhen an LLM is doing a batched decode (as in the second figure) of thousands of tokens in a single step, it’s compute bound.
\n![\"five](\"https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/batched_decode.png\")
When LLM decoding is memory bound, you’re wasting compute.\nThe GPU could be doing more operations, but it’s not.\nFor instance, if you were decoding 2 sequences instead of 1, it wouldn’t take any more time per token.
\nSpeculative decoding is a way of using that extra compute to make decoding faster (at low batch sizes).
\nTo make this concrete, let’s say the optimal number of tokens to run is 5 in a given forward pass.\nIf we ran any fewer, we’d be memory bound, any more and we’d be compute bound.
\nWe have two options.
\nFirst, we can run a batch size of 5 sequences, each decoding one at a time.\nWe’ll say this is “breadth” (across sequences):
\nAlternatively, we can use speculative decoding to run 5 tokens in one sequence, achieving high “depth” for that sequence:
\n![\"T](\"https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/depth_speculation.png\")
In both cases, the GPU is doing roughly the same amount of work.\nIn the first case, all 5 sequences make a little bit of progress, in the second, our one sequence makes a lot of progress.
\nIf you add more sequences and speculate to the same depth, you’ll just be slowing them all down as you’re compute bound.
\nBut at small batch sizes (where batch size << optimal number of tokens), speculative decoding allows you to use “free” compute to complete all the sequences faster!
\nThis might all sound a little pedantic.\nSpeculative decoding does make things faster at small batch sizes, so who cares if it’s as much or more work?
\nThis becomes important when you start to serve real-world loads or digging into speculative decoding research.\nIf the optimal batch size is 5 but we’re running with speculative decoding on, as in the following figure, we’re actually wasting compute when we could be using that time to move memory!
\n![\"five](\"https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/compute_bound_speculation.png\")
Especially if our acceptance rate isn’t 100%, every rejected token is extra wasted work.
\nThis is why, at high throughput, vLLM gives you the configurability to totally turn off speculative decoding with the --speculative-disable-by-batch-size param.
And researchers looking at this picture started to wonder: why would we speculate to the same depth at every timestep?\nWhy not speculate deep when we have extra compute (small batch size), and shallow, or even dynamically per sequence, when we have a larger batch size:
\n![\"five](\"https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast/dynamic_speculation.png\")