This is called Speculative Decoding

Gaurav Sen explains speculative decoding, the technique Google published in 2022 to make large language models answer 2 to 3 times faster with the same output: a small model drafts several tokens, the large model verifies the draft's prefixes in parallel, and the method now serves production traffic at Google and DeepSeek.

Tutorial · AI Inference · Large Language Models · Machine Learning