Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models

TL;DR

The paper addresses latency in autoregressive LLM decoding by surveying speculative execution as a draft-then-verify framework. It develops a unified taxonomy that separates drafting (source models, termination, and draft management) from verification (verifier design and acceptance criteria) and analyzes how design choices affect speedups and quality. Through critical comparison of static/adaptive drafting, exact/rejection/typical acceptance, and single versus multiple drafts, it identifies key challenges in framework design, parameter search, system integration, and objective optimization. The survey also covers practical implementations (e.g., LLaMA.cpp, vLLM) and applications (online, contrastive, on-device), outlining concrete directions for robust, scalable speculative execution in real-world LLM deployments.

Abstract

With the increasingly giant scales of (causal) large language models (LLMs), the inference efficiency comes as one of the core concerns along the improved performance. In contrast to the memory footprint, the latency bottleneck seems to be of greater importance as there can be billions of requests to a LLM (e.g., GPT-4) per day. The bottleneck is mainly due to the autoregressive innateness of LLMs, where tokens can only be generated sequentially during decoding. To alleviate the bottleneck, the idea of speculative execution, which originates from the field of computer architecture, is introduced to LLM decoding in a \textit{draft-then-verify} style. Under this regime, a sequence of tokens will be drafted in a fast pace by utilizing some heuristics, and then the tokens shall be verified in parallel by the LLM. As the costly sequential inference is parallelized, LLM decoding speed can be significantly boosted. Driven by the success of LLMs in recent couple of years, a growing literature in this direction has emerged. Yet, there lacks a position survey to summarize the current landscape and draw a roadmap for future development of this promising area. To meet this demand, we present the very first survey paper that reviews and unifies literature of speculative execution in LLMs (e.g., blockwise parallel decoding, speculative decoding, etc.) in a comprehensive framework and a systematic taxonomy. Based on the taxonomy, we present a critical review and comparative analysis of the current arts. Finally we highlight various key challenges and future directions to further develop the area.

Figures (4)

• Figure 1: An illustration of speculative execution in LLMs via a comparison between autoregressive decoding and speculative execution.
• Figure 2: A systematic taxonomy of existing literature of speculative execution in LLMs.
• Figure 3: The illustration of different categories of the drafter.
• Figure 4: The illustration of tree attention. In the tree attention, child tokens can only see their parent tokens, facilitating the parallel verification of potentially multiple drafts. In contrast, one draft would only require a chain attention, which is causal and lower-triangular.