DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting

TL;DR

DuoDecoding tackles the latency of autoregressive LLM inference by distributing the draft and target models across CPU and GPU, enabling parallel execution of drafting and verification. The method introduces a hardware-aware drafting budget and dynamic multi-sequence drafting guided by draft uncertainty to maintain draft quality while reducing idle time. Empirical results across seven tasks show substantial speedups (up to $2.61\times$) and a TTFT reduction to $83\%$ of conventional speculative decoding, with strong ablations validating the contributions. This approach demonstrates practical latency improvements for heterogeneous inference and offers a pathway to more efficient deployment of large language models on commodity hardware.

Abstract

Large language models (LLMs) exhibit exceptional performance across a wide range of tasks; however, their token-by-token autoregressive generation process significantly hinders inference speed. Speculative decoding presents a promising draft-then-verify framework that reduces generation latency while maintaining output distribution fidelity. Nevertheless, the draft model introduces additional computational overhead, becoming a performance bottleneck and increasing the time to first token (TTFT). Previous approaches to mitigate draft model overhead have primarily relied on heuristics and generally failed to match the quality of the draft language models. To address these challenges, we propose DuoDecoding, a novel approach that strategically deploys the draft and target models on the CPU and GPU respectively, enabling parallel decoding while preserving draft quality. Our method incorporates a hardware-aware optimal draft budget to minimize idle times and employs dynamic multi-sequence drafting to enhance draft quality. Extensive experiments across seven tasks show that DuoDecoding achieves up to 2.61x speedup in generation latency, while reducing TTFT to 83% of that in conventional speculative decoding. The Code is available at https://github.com/KaiLv69/DuoDecoding.

Figures (5)

• Figure 1: Wall time for draft model autoregressive generation and target model parallel verification of 8 tokens with varying input lengths. The draft phase has become a comparable bottleneck to the verification phase, and executing the lightweight draft model on CPU does not compromise generation efficiency.
• Figure 2: Dynamic multi-sequence drafting. $p_{i,j}$ represents the probability of the $j$-th ranked token at the $i$-th position in the generated sequence. $\theta= p_{1,1} \times p_{2,1}$ serves as the threshold. Tokens with probabilities $p_{1,k}$ exceeding the threshold $\theta$ will continue to be predicted sequentially, forming a independent draft sequence.
• Figure 3: Comparison of Time to First Token (TTFT) across different tasks and models. The y-axis shows the relative TTFT normalized by vanilla autoregressive generation. Lower values indicate better latency performance.
• Figure 4: Profiling of time and number of processed token in one generation iteration for different decoding strategies.
• Figure 5: Distribution of sequence numbers during generation process in DuoDecoding.