FOCUS: DLLMs Know How to Tame Their Compute Bound

TL;DR

This paper tackles the compute-bound challenge of diffusion-based large language models (DLLMs) by identifying that most of the block-wise decoding compute is wasted on non-decodable tokens. It introduces FOCUS, a training-free inference system that dynamically prioritizes decodable tokens using an attention-based predictor—the Importance Delta derived from early transformer layers—and evicts the rest to increase the effective batch size. Through dynamic budgeting and an intra-block KV cache with a neighbor-aware stability criterion, FOCUS reduces redundant FLOPs and sustains generation quality, achieving up to 2.32x real-world throughput and up to 3.52x with larger block sizes across multiple DLLMs and benchmarks. The results demonstrate robust gains with minimal overhead and establish a foundation for further decodability-driven optimization in diffusion-based decoding, enabling scalable deployment of DLLMs in production settings.

Abstract

Diffusion Large Language Models (DLLMs) offer a compelling alternative to Auto-Regressive models, but their deployment is constrained by high decoding cost. In this work, we identify a key inefficiency in DLLM decoding: while computation is parallelized over token blocks, only a small subset of tokens is decodable at each diffusion step, causing most compute to be wasted on non-decodable tokens. We further observe a strong correlation between attention-derived token importance and token-wise decoding probability. Based on this insight, we propose FOCUS -- an inference system designed for DLLMs. By dynamically focusing computation on decodable tokens and evicting non-decodable ones on-the-fly, FOCUS increases the effective batch size, alleviating compute limitations and enabling scalable throughput. Empirical evaluations demonstrate that FOCUS achieves up to 3.52$\times$ throughput improvement over the production-grade engine LMDeploy, while preserving or improving generation quality across multiple benchmarks. The FOCUS system is publicly available on GitHub: https://github.com/sands-lab/FOCUS.

Figures (15)

• Figure 1: Comparison of Inference Paradigms. (Left) ARLLMs generate tokens one token at a time. (Right) DLLMs process an entire block in parallel, yet decode only a subset at every step.
• Figure 2: Distribution of decoded tokens per step across benchmarks. Using the SDAR-8B-Chat model with block size $B=32$ and $confidence\_threshold=0.9$, the data reveals that the mean proportion of successfully decoded tokens is typically only around $10\%$, indicating that $\sim90\%$ of the block-wise computation is redundant. Please see Appendix \ref{['appendix_decoding_stats']} for more results.
• Figure 3: Layer-wise Importance. (Top) Decoded tokens dominate. (Bottom) Filtering them reveals Decodable tokens diverge from Non-Decodable from Layer 1. Settings match Figure \ref{['fig:decoding_stats']}.
• Figure 4: Importance Delta vs. Decodability. Decodable tokens (green) cluster at high deltas, distinct from Non-Decodable ones (grey). Settings match Figure \ref{['fig:decoding_stats']}. See Appendix \ref{['appendix_decodability']} for more results.
• Figure 5: FOCUS design overview. The workflow centers on (Middle)Token Eviction, which performs early filtering after the Layer 1 Q/K projections (QKProj). It prioritizes decodable token candidates based on the Importance Delta ($\Delta\mathcal{I}$ in Eq. \ref{['eq:delta']}) from (Right)Delta Calculation, and retaining only the top candidates within the adaptive token budget determined by (Left)Dynamic Budgeting.

Theorems & Definitions (1)

• Proposition 4.1: Exponential Decay of Eviction Error