ConFu: Contemplate the Future for Better Speculative Sampling

TL;DR

This work believes its work is the first to bridge speculative decoding with continuous reasoning tokens, offering a new direction for accelerating LLM inference.

Abstract

Speculative decoding has emerged as a powerful approach to accelerate large language model (LLM) inference by employing lightweight draft models to propose candidate tokens that are subsequently verified by the target model. The effectiveness of this paradigm critically depends on the quality of the draft model. While recent advances such as the EAGLE series achieve state-of-the-art speedup, existing draft models remain limited by error accumulation: they condition only on the current prefix, causing their predictions to drift from the target model over steps. In this work, we propose \textbf{ConFu} (Contemplate the Future), a novel speculative decoding framework that enables draft models to anticipate the future direction of generation. ConFu introduces (i) contemplate tokens and soft prompts that allow the draft model to leverage future-oriented signals from the target model at negligible cost, (ii) a dynamic contemplate token mechanism with MoE to enable context-aware future prediction, and (iii) a training framework with anchor token sampling and future prediction replication that learns robust future prediction. Experiments demonstrate that ConFu improves token acceptance rates and generation speed over EAGLE-3 by 8--11% across various downstream tasks with Llama-3 3B and 8B models. We believe our work is the first to bridge speculative decoding with continuous reasoning tokens, offering a new direction for accelerating LLM inference.

Figures (4)

• Figure 1: Illustration of the purpose of future generation direction prediction
• Figure 2: Overview of ConFu's inference pipeline. Given the input tokens, the target model first produces the next output token along with a future prediction vector $\bm{f}$, using both prompt tokens and contemplate tokens. The draft model then conditions on $\bm{f}$ as an additional future token to autoregressively generate draft tokens. Throughout the drafting process, the future token $\bm{f}$ remains fixed and is always appended to the end of the input sequence.
• Figure 3: Verification with contemplate tokens in ConFu. Let $t_1,t_2,t_3$ denote draft tokens in the speculative tree. We insert one contemplate token after each draft token so that the target model can simultaneously verify draft candidates and generate the corresponding future predictions. The tree attention mask is adjusted accordingly to ensure correct verification and alignment of future predictions with accepted tokens.
• Figure 4: Illustration of Dynamic Contemplate Tokens with MoE. The input tokens contain both accepted tokens and the draft tokens of the current iteration. The MoE module only takes the hidden representation of the last accepted token as inputs. Then it computes the expert weights with a linear layer (router) and outputs the weighted sum of the selected learnable embeddings as the final contemplate token embedding. For simplicity a single [con] token is shown instead of 3 (1 for 'like' and 2 for draft tokens)