Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelism

TL;DR

The paper tackles memory-bound inference in autoregressive LLMs by addressing the limitations of existing Speculative Decoding approaches. It introduces Double Retrieval Speculative Parallelism (Double), a training-free framework that combines iterative draft-side retrieval with target-side multi-token verification to break the theoretical speedup ceiling and reduce mid-sequence rejections. By incorporating a hierarchical datastore and target-guided forward guidance, Double expands draft length beyond latency constraints while preserving the target distribution, achieving lossless acceleration. Theoretical contributions include extending the speedup analysis to multi-round decoding and proving a new bound $C(1 + \text{AMT})$, along with practical demonstrations across multiple models and benchmarks showing up to $5.3\times$ speedups. The work offers a scalable, data-efficient pathway to accelerate LLM inference without retraining, with clear guidance for industrial deployment and engineering optimizations.

Abstract

Parallel Speculative Decoding (PSD) accelerates traditional Speculative Decoding (SD) by overlapping draft generation with verification. However, it remains hampered by two fundamental challenges: (1) a theoretical speedup ceiling dictated by the speed ratio between the draft and target models, and (2) high computational waste and pipeline stall due to mid-sequence token rejections of early errors. To address these limitations, we introduce \textsc{Double} (Double Retrieval Speculative Parallelism). By bridging the gap between SD and PSD, our framework resolves the Retrieval \emph{Precision-Efficiency Dilemma} through a novel synchronous mechanism. Specifically, we enable the draft model to execute iterative retrieval speculations to break the theoretical speedup limits; to alleviate rejections without rollback, the target model performs authoritative retrieval to generate multi-token guidance. \textsc{Double} is entirely training-free and lossless. Extensive experiments demonstrate state-of-the-art speedup of $\textbf{5.3}\times$ on LLaMA3.3-70B and $\textbf{2.8}\times$ on Qwen3-32B, significantly outperforming the advanced method EAGLE-3 that requires extensive model training.

Figures (8)

• Figure 1: Comparison between SD, PSD and Double. (a) SD suffers from pipeline bubbles due to sequential dependency. (b) PSD overlaps these processes to reduce latency but struggles with mid-sequence rejection, where tokens generated are wasted after an early error (e.g., red boxes $x_{6-9}$). (c)Double resolves these issues through a double-retrieval mechanism: draft model leverages retrieval to expand draft length (more tokens $x_{1\text{-}5}$); target model performs retrieval to offer multi-token pre-verify and guidance (retrieve $y_{1\text{-}3}$ for verifying $x_{1\text{-}3}$) to ensure precision, thereby breaking the speed limit $C$ jointly and mitigating rejection penalties.
• Figure 2: Speedup ratios of different methods on HumanEval and CNN/DM. Double achieves a speedup of 5.3$\times$ on LLama3.3-70B and 2.8$\times$ on Qwen3-32B over than EAGLE3. Full results are available in Table \ref{['tab:main_resultes']}.
• Figure 3: Motivation of Double. (a) Breaking the theoretical speedup ceiling $C$ across five model pairs on three benchmarks. Green regions indicate where Double surpasses the speedup limit. (b) Retrieval precision-efficiency trade-off comparison on Deepseek-1.3B&33B, showing Double achieves optimal balance between effective matched tokens and speedup compared to draft-side (Ouroboros) and target-side (PLD, Token Recycling) methods.
• Figure 4: Workflow of Double.(a) Retrieval Unit: Utilizes hierarchical datastores to propose $d$ candidates with match length $s$. (b) Step $T_1$: At speed ratio $C=3$, $\mathcal{M}_q$ executes iterative retrieval to draft 5 tokens, while $\mathcal{M}_p$ provide the multi-tokens pre-verify. (c) Step $T_2$: Target retrieval rectifies $x_{7\text{-}9}$ ("for submitting novel" $\to$ "in the field") as a Correction. (d) Step $T_3$: Target retrieval directly extends the sequence beyond the draft as an Extension.
• Figure 5: Ablation study on LLaMA-3.3-70B: both retrieval components are indispensable for achieving optimal speedup and accepted length.