Speculative Decoding

What Is Speculative Decoding?

Normally a large model generates one token at a time, which is slow. Speculative decoding uses a small draft model to propose several next tokens, then the large model checks them in a single forward pass and accepts the ones that match what it would have produced. Accepted tokens come almost free, so latency drops with no quality loss because the large model still has final say. The gain depends on how often the draft model guesses right.