Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

TL;DR

A comprehensive survey of speculative decoding methods, categorizing them into draft-centric and model-centric approaches and discussing key ideas associated with each method, highlighting their potential for scaling LLM inference.

Abstract

Efficient inference in large language models (LLMs) has become a critical focus as their scale and complexity grow. Traditional autoregressive decoding, while effective, suffers from computational inefficiencies due to its sequential token generation process. Speculative decoding addresses this bottleneck by introducing a two-stage framework: drafting and verification. A smaller, efficient model generates a preliminary draft, which is then refined by a larger, more sophisticated model. This paper provides a comprehensive survey of speculative decoding methods, categorizing them into draft-centric and model-centric approaches. We discuss key ideas associated with each method, highlighting their potential for scaling LLM inference. This survey aims to guide future research in optimizing speculative decoding and its integration into real-world LLM applications.

Figures (3)

• Figure 1: Taxonomy of various speculative decoding methods
• Figure 2: On the left is a general draft-centric implementation that shows the focus on selecting from a smaller pool of drafted tokens compared to standard speculative decoding methods. On the right is a general model-centric implementation that shows that a refined drafting model is used to create better, higher quality initial draft outputs.
• Figure 3: Timeline of the various speculative decoding methods discussed in this paper