When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding

TL;DR

ReSpec addresses the Drafter's Dilemma in retrieval-enhanced speculative decoding by introducing entropy-guided triggering, feedback-driven candidate selection, and source-aware relaxed verification to adaptively manage retrieval. The approach treats drafting sources as a dynamic, context-dependent choice rather than a fixed heuristic, achieving state-of-the-art acceleration on Spec-Bench with consistent quality across Vicuna-7B and Qwen-7B. Key contributions include formalizing the adaptive trigger, EMA-based candidate scoring, and a look-ahead tolerant verification strategy, validated by comprehensive experiments and ablations. The results demonstrate practical, robust speedups for large language model inference, enabling faster, more responsive generation with controlled accuracy.

Abstract

Speculative decoding (SD) has emerged as an effective technique to accelerate large language model (LLM) inference without compromising output quality. However, the achievable speedup largely depends on the effectiveness of the drafting model. While model-based methods like EAGLE-2 are accurate but costly, retrieval-enhanced methods like SAM-Decoding rely on heuristic switching strategies that often trigger unnecessary retrievals. To address this, we propose ReSpec (\textbf{Re}trieval-enhanced \textbf{Spe}culative Decoding), a novel framework that transforms heuristic drafter switching into adaptive decision-making. ReSpec features three core innovations: 1) An \textbf{entropy-guided adaptive trigger} quantifies contextual predictability to initiate retrieval only when uncertainty is low, avoiding costly low-quality speculations. 2) A \textbf{feedback-driven candidate selection} leverages historical feedback to organize multiple high-quality candidates for parallel verification, maximizing retrieval utility. 3) A source-aware \textbf{relaxed verification strategy} applies strict checks to model-generated drafts while using a relaxed verification for retrieved drafts, achieving a better balance between accuracy and efficiency. Extensive experiments on Spec-Bench demonstrate that ReSpec achieves state-of-the-art acceleration,outperforming EAGLE-2 and SAM-Decoding by over $33\%$ and $25\%$, respectively, while maintaining output quality.

Figures (4)

• Figure 1: Overview of three key limitations in current hybrid speculative decoding methods and our proposed solutions. (a) Motivating example from document summarization showing how current methods fail with ambiguous triggers while our framework succeeds through entropy-guided selection. (b) Detailed workflow of our three-phase approach. Phase 1 evaluates $k=3$ suffix candidates using confidence score $C = H + \lambda/\textit{length}$ where $H$ is the entropy of next-token distribution. Phase 2 filters retrieved positions based on EMA quality scores $S_p$ updated via $S_p \leftarrow (1-\alpha)S_p + \alpha R_p$ where $R_p$ is the historical acceptance rate. Phase 3 applies hybrid verification: relaxed (top-$\beta$ with tolerance $\tau$) for retrieval drafts and strict for model-generated drafts.
• Figure 2: Relative speedup on Spec-Bench tasks. ReSpec demonstrates strong and balanced performance across all domains, outperforming specialized methods.
• Figure 3: Effect of $\tau$ on MAT and GPT-4o quality scores for Sum task using Qwen2-7B.
• Figure 4: Effect of $m$ on MAT and GPT-4o quality scores for Sum task using Qwen2-7B.