Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs

TL;DR

This work tackles the latency–quality tension of diffusion LLMs (dLLMs) when used as drafts in speculative decoding by proposing FailFast, a dynamic drafting framework that leverages dLLMs as ultra-fast but error-prone drafters. FailFast minimizes draft compute in hard regions (fail fast) and aggressively extends speculation in easy regions (win big), guided by a confidence signal to reduce verification rounds with autoregressive verifiers. Key contributions include a practical algorithm for adaptive speculation length, a confidence-based difficulty signal, and end-to-end speedups up to $4.9\times$ over vanilla decoding without any fine-tuning, validated across multiple benchmarks and target models. The approach demonstrates that dLLMs are well-suited for drafting in speculative decoding when paired with runtime adaptation, offering a significant practical impact for low-latency LLM inference. The work is complemented by an open-source release to enable replication and broader adoption.

Abstract

Diffusion Large Language Models (dLLMs) offer fast, parallel token generation, but their standalone use is plagued by an inherent efficiency-quality tradeoff. We show that, if carefully applied, the attributes of dLLMs can actually be a strength for drafters in speculative decoding with autoregressive (AR) verifiers. Our core insight is that dLLM's speed from parallel decoding drastically lowers the risk of costly rejections, providing a practical mechanism to effectively realize the (elusive) lengthy drafts that lead to large speedups with speculative decoding. We present FailFast, a dLLM-based speculative decoding framework that realizes this approach by dynamically adapting its speculation length. It "fails fast" by spending minimal compute in hard-to-speculate regions to shrink speculation latency and "wins big" by aggressively extending draft lengths in easier regions to reduce verification latency (in many cases, speculating and accepting 70 tokens at a time!). Without any fine-tuning, FailFast delivers lossless acceleration of AR LLMs and achieves up to 4.9$\times$ speedup over vanilla decoding, 1.7$\times$ over the best naive dLLM drafter, and 1.4$\times$ over EAGLE-3 across diverse models and workloads. We open-source FailFast at https://github.com/ruipeterpan/failfast.

Figures (9)

• Figure 1: Intuition behind FailFast and other baseline drafters. The AR drafter takes 10 forward passes to speculate 10 tokens, whereas the naive dLLM drafter employs confidence-aware parallel decoding, taking $<10$ forward passes to reach a similar quality. In contrast, FailFast spends minimal compute and dynamically determines how many tokens to propose based on the confidence of speculated tokens. In harder-to-speculate regions (top right), it adopts a shorter speculation length and minimizes the amount of compute to "fail fast", further minimizing the speculation latency. In easier-to-speculate regions (bottom right), it aggressively extends the speculation length to "win big" and reduce the verification overhead. Note that this example showcases two extreme cases of FailFast, whereas it dynamically navigates through a spectrum of speculation lengths based on decoding difficulty at runtime.
• Figure 2: dLLM's concavity of accuracy improvements when speculating 8 tokens per round. Doing 2.6$\times$ more drafter forward passes only increased the acceptance rate from 53.2% to 60.6%.
• Figure 3: The varying difficulty of decoding within a sequence. The raster plot visualizes whether each output token is accepted (green, "easier") or rejected and regenerated by the target model (white, "harder"). Note that an all-white chunk at the right end indicates a shorter sequence.
• Figure 4: Breakdown of end-to-end latency into speculation (lighter, upper) and verification (darker, lower). While applying a dLLM drafter (Fast-dLLM) reduces the speculation latency, FailFast further reduces the speculation latency (in each round) while also reducing the verification latency (the number of rounds).
• Figure 5: CDF of the number of accepted/speculated tokens in each round for Qwen2.5-32B on MATH. Full data are in Tab. \ref{['tab:performance_details']}.