Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules

Amr Mohamed, Yang Zhang, Michalis Vazirgiannis, Guokan Shang

TL;DR

SchED reframes diffusion decoding as a stopping-time problem by applying a progress-conditioned, smooth threshold on a full-span confidence measure, enabling training-free early exits across diffusion-LM architectures. It is model-agnostic and integrates with existing transfer schedules, delivering large speedups with minimal quality loss on instruction-tuned models and consistent gains on base models. Empirical results show up to 4x speedups with near-parity accuracy and superior quality–speed trade-offs under the QPS metric, supported by entropy analyses that explain confidence stabilization dynamics. This approach offers a practical, scalable path to deploying diffusion-based language models in latency-sensitive and throughput-constrained settings.

Abstract

Diffusion large language models (dLLMs) offer a promising alternative to autoregressive models, but their practical utility is severely hampered by slow, iterative sampling. We present SchED, a training-free, model-agnostic early-exit algorithm that aggregates full-span logit margins and halts decoding once a smooth, progress-dependent confidence threshold is met. We evaluated SchED on two dLLM families (Dream and LLaDA), in base and instruction-tuned variants across ten benchmarks spanning downstream tasks including multiple-choice question answering (MCQ), math, long-form QA/summarization, and translation. SchED delivers large, stable accelerations: on instruction-tuned models, it achieves $3.8$-$4.0\times$ speedups while retaining $99.8$-$100\%$ of the baseline score on average. On base models, SchED yields consistent speedup gains with $99.1$-$100\%$ performance retention, with up to $2.34\times$ under more aggressive settings. Using a conservative speed metric that heavily penalizes quality loss (QPS, $γ{=}4$), we show that SchED is robust and clearly outperforms prior confidence-based early-exit methods, which break down on long-form generation. An entropy analysis of the model's token predictions reveals that instruction tuning speeds up the decay of predictive entropy. By turning genuine confidence stabilization into computational savings, SchED makes dLLM decoding substantially more efficient.

Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules

TL;DR

SchED reframes diffusion decoding as a stopping-time problem by applying a progress-conditioned, smooth threshold on a full-span confidence measure, enabling training-free early exits across diffusion-LM architectures. It is model-agnostic and integrates with existing transfer schedules, delivering large speedups with minimal quality loss on instruction-tuned models and consistent gains on base models. Empirical results show up to 4x speedups with near-parity accuracy and superior quality–speed trade-offs under the QPS metric, supported by entropy analyses that explain confidence stabilization dynamics. This approach offers a practical, scalable path to deploying diffusion-based language models in latency-sensitive and throughput-constrained settings.

Abstract

Diffusion large language models (dLLMs) offer a promising alternative to autoregressive models, but their practical utility is severely hampered by slow, iterative sampling. We present SchED, a training-free, model-agnostic early-exit algorithm that aggregates full-span logit margins and halts decoding once a smooth, progress-dependent confidence threshold is met. We evaluated SchED on two dLLM families (Dream and LLaDA), in base and instruction-tuned variants across ten benchmarks spanning downstream tasks including multiple-choice question answering (MCQ), math, long-form QA/summarization, and translation. SchED delivers large, stable accelerations: on instruction-tuned models, it achieves - speedups while retaining - of the baseline score on average. On base models, SchED yields consistent speedup gains with - performance retention, with up to under more aggressive settings. Using a conservative speed metric that heavily penalizes quality loss (QPS, ), we show that SchED is robust and clearly outperforms prior confidence-based early-exit methods, which break down on long-form generation. An entropy analysis of the model's token predictions reveals that instruction tuning speeds up the decay of predictive entropy. By turning genuine confidence stabilization into computational savings, SchED makes dLLM decoding substantially more efficient.

Paper Structure

This paper contains 29 sections, 14 equations, 1 figure, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion LLMs.
  4. Sampling efficiency and early commit.
  5. Methods
  6. Preliminaries: Discrete Diffusion Language Models
  7. Setup and notation.
  8. Forward (masking) process $q$.
  9. Reverse (denoising) process $p_\theta$.
  10. Masked diffusion with partial unmasking (transfer schedule).
  11. Training objective.
  12. SchED: Schedule-based Early Decoding
  13. Confidence measurement.
  14. Early-exit trigger.
  15. Smooth threshold schedules.
  16. ...and 14 more sections

Figures (1)

  • Figure 1: Mean predictive entropy across diffusion steps for five input types (GPQA, GSM8K, HotpotQA, MMLU, WMT14 EN$\rightarrow$FR). Curves show per-token entropy (nats/token) averaged over evaluation samples; shaded bands denote one standard deviation across samples. Both Dream Base and Dream Instruct exhibit monotonically decreasing entropy as denoising progresses.