Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding

TL;DR

Streaming-dLLM tackles the slow inference of diffusion LLMs by addressing spatial suffix redundancy and temporal inefficiency in block-wise diffusion. It introduces Attenuation Guided Suffix Modeling to prune redundant suffix attention and Dynamic Confidence Aware Parallel Decoding with an EOS-based Early Exit, all in a training-free framework. Across Dream and LLaDA backbones and multiple benchmarks, it achieves up to 68.2x throughput improvement with comparable or improved generation quality, and substantial reductions in per-sample latency. The approach significantly enhances practical deployment of diffusion decoding for long-context generation.

Abstract

Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation, leveraging parallel decoding and bidirectional attention to achieve superior global coherence compared to autoregressive models. While recent works have accelerated inference via KV cache reuse or heuristic decoding, they overlook the intrinsic inefficiencies within the block-wise diffusion process. Specifically, they suffer from spatial redundancy by modeling informative-sparse suffix regions uniformly and temporal inefficiency by applying fixed denoising schedules across all the decoding process. To address this, we propose Streaming-dLLM, a training-free framework that streamlines inference across both spatial and temporal dimensions. Spatially, we introduce attenuation guided suffix modeling to approximate the full context by pruning redundant mask tokens. Temporally, we employ a dynamic confidence aware strategy with an early exit mechanism, allowing the model to skip unnecessary iterations for converged tokens. Extensive experiments show that Streaming-dLLM achieves up to 68.2X speedup while maintaining generation quality, highlighting its effectiveness in diffusion decoding. The code is available at https://github.com/xiaoshideta/Streaming-dLLM.

Figures (14)

• Figure 1: Comparison of accuracy and throughput across different acceleration strategies. Our proposed method improves inference throughput while maintaining competitive accuracy compared to prior approaches.
• Figure 2: Attention distribution at the final layer (Layer 31) of LLaDA-1.5, showing interactions between the current block and the full input sequence. The solid line denotes the mean attention score at each diffusion step, the shaded region represents the interquartile range (IQR, 25%–75%), and gray dashed lines delineate the prefix, current-generation and suffix regions. Attention is concentrated on a few neighboring suffix blocks and the final token and most intermediate suffix positions receive negligible attention, highlighting redundancy in the suffix during block generation.
• Figure 3: Token confidence distribution during iterative block-wise generation of LLaDA-1.5 on GSM8K with a generation length of 256, showing that mean token confidence steadily increases over iterations. The solid line denotes the mean confidence at each diffusion step and the shaded area represents the interquartile range (IQR, 25%--75%). While a high confidence threshold ensures generation quality, it can be conservative, potentially delaying the acceptance of sufficiently confident tokens. Visualizations of additional blocks are provided in the \ref{['sec:Details of Attention']}.
• Figure 4: Illustration of Attenuation-Guided Suffix Modeling. For each block, the nearest neighboring region following the current block is retained using a sliding window (red dashed box) and concatenated with the trailing position to form an approximate suffix region.
• Figure 5: Ablation study on the sliding window size (no suffix windows, mean $size=512$).