PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length

TL;DR

This paper tackles the latency of autoregressive LLM inference by addressing the inefficiency of fixed draft window sizes in speculative decoding. It introduces Pacer, a blockwise, trainable pre-verification module that dynamically controls draft length by evaluating blockwise acceptance probabilities and halting drafts when predicted acceptance falls below a threshold, with the threshold growing over time. Empirical results across multiple model families and tasks show Pacer achieving up to $2.66\times$ speedups over autoregressive decoding and up to $3.09\times$ when combined with Ouroboros, consistently outperforming baselines and other dynamic-draft-length methods. Ablation studies validate the importance of draft-position embeddings and the growth factor, and the approach proves compatible with existing SD methods, underscoring its practical impact for accelerating large-language-model inference.

Abstract

Speculative decoding (SD) is a powerful technique for accelerating the inference process of large language models (LLMs) without sacrificing accuracy. Typically, SD employs a small draft model to generate a fixed number of draft tokens, which are then verified in parallel by the target model. However, our experiments reveal that the optimal draft length varies significantly across different decoding steps. This variation suggests that using a fixed draft length limits the potential for further improvements in decoding speed. To address this challenge, we propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer. This layer pre-verifies draft tokens blockwise before they are sent to the target model, allowing the draft model to stop token generation if the blockwise pre-verification fails. We implement Pacer on multiple SD model pairs and evaluate its performance across various benchmarks. Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding. Furthermore, when integrated with Ouroboros, Pacer attains up to 3.09x Speedup.

Figures (6)

• Figure 1: Comparison of the speculative decoding (SD) process for Pacer and vanilla SD with fixed small and large window sizes. (a) Pacer generates drafts in blocks of size $b=3$, performs three rounds of pre-verification, and produces a total of 9 draft tokens, of which 7 are accepted with only one target model forward pass. (b) Vanilla SD with a small window size ($\gamma=2$) causes the draft model to stop prematurely, resulting in 3 costly target forward and 6 draft forward. (c) Vanilla SD with a large window size $\gamma=9$ generates 9 draft tokens, but only 2 are accepted, leading to wasted draft computation.
• Figure 2: (a) Maximum acceptance lengths across decoding steps. The optimal fixed window size ($\gamma=9$) is marked by the red horizontal line. Significant variation in acceptance lengths highlights the inefficiency of employing a fixed draft window size. (b) Comparison between speculative decoding using the optimal fixed window ($\gamma=9$) and optimal dynamic window size ($\gamma^\star$). Utilizing dynamic draft lengths substantially reduces forward passes for both draft and target models, leading to increased decoding speed (tokens/s).
• Figure 3: Comparison of decoding speeds (tokens/s) and speedup for different methods on the HumanEval dataset. Pacer consistently outperforms all baseline methods across various models.
• Figure 4: An illustrative comparison between vanilla SD with a fixed window size ($\gamma = 6$) and Pacer with a block size of $b = 3$. The top part of the figure shows the accelerated token generation achieved by each method. The bottom part presents a latency breakdown for generating the code snippet Fibonacci(n): if n == 0.
• Figure 5: Customized attention mask for packing multiple draft steps into a single sequence for efficient training.