DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference

TL;DR

DART targets the drafting bottleneck in speculative decoding by adopting a diffusion-inspired approach that predicts multiple future logits in parallel from the target model's hidden states. It combines a lightweight, prefix-conditioned draft model with shifted logits prediction and an efficient, continuity-aware N-gram tree pruning to construct a high-quality draft token tree for verification, eliminating autoregressive drafting rollout. Training uses prefix-shared masked objectives and an annealed KL divergence to align draft distributions with the target model, enabling stable, parallel supervision. Empirically, DART achieves 2.03×–3.44× end-to-end speedups and up to 6.8× drafting-forward speedups over prior methods like EAGLE3, with code available to reproduce and deploy this faster speculative decoding framework.

Abstract

Speculative decoding is an effective and lossless approach for accelerating LLM inference. However, existing widely adopted model-based draft designs, such as EAGLE3, improve accuracy at the cost of multi-step autoregressive inference, resulting in high drafting latency and ultimately rendering the drafting stage itself a performance bottleneck. Inspired by diffusion-based large language models (dLLMs), we propose DART, which leverages parallel generation to reduce drafting latency. DART predicts logits for multiple future masked positions in parallel within a single forward pass based on hidden states of the target model, thereby eliminating autoregressive rollouts in the draft model while preserving a lightweight design. Based on these parallel logit predictions, we further introduce an efficient tree pruning algorithm that constructs high-quality draft token trees with N-gram-enforced semantic continuity. DART substantially reduces draft-stage overhead while preserving high draft accuracy, leading to significantly improved end-to-end decoding speed. Experimental results demonstrate that DART achieves a 2.03x--3.44x wall-clock time speedup across multiple datasets, surpassing EAGLE3 by 30% on average and offering a practical speculative decoding framework. Code is released at https://github.com/fvliang/DART.

Figures (7)

• Figure 1: Average Acceptance Length ($\tau$) versus drafting forward latency (ms), averaged across all benchmarks, for speculative decoding of Qwen3-14B on H20-3e GPU. Compared with EAGLE3 and SPS (standard speculative sampling; using Qwen3-1.7B as the draft model with a draft length of 5, set to yield a measurable speedup, whereas both EAGLE3 and DART use a draft length of 8), DART reduces drafting forward latency by up to $6.8\times$ and $53.3\times$, respectively, while preserving relatively high $\tau$, demonstrating a significantly improved drafting efficiency.
• Figure 2: Speedup over vanilla autoregressive decoding (batch size $=1$), averaged across all datasets. For Qwen3 models, results are reported at temperature $T=0$ on Qwen3-1.7B, 4B, 8B, 14B, 32B, and additionally at $T=1$ on Qwen3-14B, 32B; only DART and EAGLE3 are compared on Qwen3, except for Qwen3-32B at $T=0$, where we also include SPS (using Qwen3-1.7B as the draft model with draft length 5). For LLaMA2-Chat-7B, we compare DART with methods including Medusa, Lookahead, SPS, and PLD.
• Figure 3: Diagram of the DART inference pipeline, illustrating the three substeps of DART's speculative decoding. $\mathbf{l}, \mathbf{m}, \mathbf{h}$ represent the low, middle, and high-level features of the target model, respectively. $\mathbf{e}$ denotes the embedding. Unlocked icon means learnable parameter. After feature extraction, we append $(d-1)$ Mask tokens to the prefix and conduct single forward to get $d$ logits, where the first logit comes from the output of the last position in prefix. We call this "Shifted logits prediction" in Section \ref{['sec:dart_draft']}. During "Continuity-Aware Tree Pruning", candidate tokens are selected from the corresponding position's predicted logit and a N-gram model ensures that the expanded tokens maintain continuity. After getting the final pruned draft token tree, we verify them in target model with Tree Attention.
• Figure 4: Position IDs and attention mask during prefix-share training. The attention mask combines clean data causal attention (Prompt Causal), prefix attention for every mask block (Mask Preceding), and block-inner causal attention (Mask Causal). For clarity of presentation, the figure depicts a simplified example with a prefix length of 3 and mask block length of 2, rather than mask block length of 7 in the actual DART training.
• Figure 5: Latency of verification, drafting forward and tree search in one draft-verify iteration when accelerating Qwen3-14B using SPS, EAGLE3 and DART. SPS has slightly lower latency in verification because of fewer draft tokens than EAGLE3 and DART.