Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge

TL;DR

Clover addresses the inefficiency of auto-regressive decoding on GPUs by integrating sequential knowledge into speculative decoding. It extends the Medusa framework with three components—Regressive Connection, Attention Decoder, and Augmenting Block—to leverage sequential information from previously speculated tokens and the full input, improving the hit rate of speculators and producing denser token trees. Empirical results on Baichuan models show Clover achieving up to $2.56\times$ throughput over vanilla decoding and $1.25$–$1.43\times$ over Medusa, with $11.7\%$–$26.4\%$ gains in speculative-head accuracy and $50\%$–$76\%$ more extra tokens per step. This approach offers a practical path to higher throughput for large LLMs in real-time serving, especially at large batch sizes and with modest token-tree sizes, by effectively exploiting sequential dependencies during speculation.

Abstract

Large language models (LLMs) suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically, billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation. Recently, parallel decoding, a type of speculative decoding algorithms, is becoming more popular and has demonstrated impressive efficiency improvement in generation. It introduces extra decoding heads to large models, enabling them to predict multiple subsequent tokens simultaneously and verify these candidate continuations in a single decoding step. However, this approach deviates from the training objective of next token prediction used during pre-training, resulting in a low hit rate for candidate tokens. In this paper, we propose a new speculative decoding algorithm, Clover, which integrates sequential knowledge into the parallel decoding process. This enhancement improves the hit rate of speculators and thus boosts the overall efficiency. Clover transmits the sequential knowledge from pre-speculated tokens via the Regressive Connection, then employs an Attention Decoder to integrate these speculated tokens. Additionally, Clover incorporates an Augmenting Block that modifies the hidden states to better align with the purpose of speculative generation rather than next token prediction. The experiment results demonstrate that Clover outperforms the baseline by up to 91% on Baichuan-Small and 146% on Baichuan-Large, respectively, and exceeds the performance of the previously top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on Baichuan-Large, respectively.

Figures (8)

• Figure 1: Overview of Medusa decoding and our extended Clover Decoding.
• Figure 2: Throughput on a model with approximately 30B parameters, supposing speculation length is 5 with 0.4 acceptance rate.
• Figure 3: The comparison between Auto-regressive Decoding and Speculative Decoding. Speculative Decoding may generate multiple tokens in a single step based on the speculation, thus achieves less decoding iteration and lower inference latency.
• Figure 4: A demonstration of Tree Attention in Speculative Decoding. Multiple speculations are merged by prefix matching to form a tree, and its topology dependency is represented in a 2-D matrix as the casual mask in Attention computation.
• Figure 5: Detailed architecture design of Clover.