Accelerate Speculative Decoding with Sparse Computation in Verification

TL;DR

This work tackles the high computational cost of speculative decoding by targeting the verification stage, identifying structured redundancy across attention, FFN, and MoE components. It proposes a unified sparse verification framework that sparsifies multiple modules during verification, augmented by inter-layer retrieval reuse and adaptive expert skipping, all without retraining. The approach yields substantial FLOPs reductions and favorable efficiency-accuracy trade-offs across long-context, QA, and math benchmarks, while maintaining stable acceptance lengths. These findings enable practical acceleration of autoregressive inference for large-scale LLMs with minimal degradation in generation quality.

Abstract

Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel. However, the verification stage often becomes the dominant computational bottleneck, especially for long-context inputs and mixture-of-experts (MoE) models. Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding to remove substantial computational redundancy in LLMs. This work systematically adopts different sparse methods on the verification stage of the speculative decoding and identifies structured redundancy across multiple dimensions. Based on these observations, we propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost. The framework further incorporates an inter-draft token and inter-layer retrieval reuse strategy to further reduce redundant computation without introducing additional training. Extensive experiments across summarization, question answering, and mathematical reasoning datasets demonstrate that the proposed methods achieve favorable efficiency-accuracy trade-offs, while maintaining stable acceptance length.

Figures (2)

• Figure 1: An illustration of the three sparse verification strategies within a single Transformer layer. Residual connections and normalization layers are omitted for clarity. A KV block contains the key-value states of multiple prefix tokens. In sparse attention, the selection of blocks is determined by the dot product between the query of the first draft token and a representative value of each block. Multiple draft tokens select the same set of blocks. In sparse FFN, the pruned channels are determined by the output of the activation function and a given threshold. In sparse MoE, a dynamic strategy based on the weights of the selected experts determines whether an expert participates in computation, and different draft tokens may involve different numbers of experts.
• Figure 2: The overlap ratio of the KV block selection for the draft token pairs with different position distances.