Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

TL;DR

Speculative Decoding (SpecDec) introduces Spec-Drafter and Spec-Verification to accelerate seq2seq autoregressive decoding via speculative execution. By deploying an independent drafting model and a relaxed verification criterion, SpecDec achieves approximately 4.6×–5.5× speedups on Transformer-based MT tasks with generation quality on par with beam search, while preserving the target model's behavior. The work provides design principles for drafting and verification, demonstrates practical advantages in latency-throughput trade-offs and adaptability to pretrained models, and situates itself within the broader speculative decoding literature. It also discusses memory-time trade-offs and real-world deployment implications, suggesting SpecDec as a viable de facto standard for efficient Transformer decoding.

Abstract

We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around $5\times$ speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

Figures (6)

• Figure 1: Compared with autoregressive decoding (left) that generates token by token, the draft-then-verify paradigm (right) first drafts multiple tokens efficiently and then verifies these tokens in parallel. Drafted tokens after the bifurcation position (e.g., $\widetilde{y}_5$) will be discarded to guarantee the generation quality.
• Figure 2: (a) Blockwise Decoding that introduces $k-1$ FFN heads on top of the target AR model for drafting the next $k$ tokens with shared attention; (b) Spec-Drafter is an independent model for drafted token prediction. It employs distinct attention queries for predicting each drafted token. Modules colored in yellow belong to the original AR model while those colored in red denote newly introduced modules.
• Figure 3: Upper: An AR model's attention heatmap showing that different target positions should attend to different source tokens; Lower: The Spec-Drafter's attention heatmap showing its capability of modeling drafted tokens in different positions, which highly aligns with the AR counterpart.
• Figure 4: Illustration of Spec-Verification. Compared to the vanilla verification strategy strictly requiring the drafted tokens to match the AR top-1 result, Spec-Verification slightly relaxes the criterion to trust the drafts more, by only requiring the drafted tokens to fall in the top-$\beta$ AR candidates with a tolerable log-likelihood gap (not shown in this Figure; see Eq (\ref{['eq:SpecDec++2']})). As a result, Spec-Verification allows more drafted tokens to be accepted even if they are slightly different from the AR top-1 result, leading to a higher inference speedup.
• Figure 5: The latency-throughput curve with various batch sizes on WMT14 EN$\rightarrow$DE.