Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow

TL;DR

This paper systematically analyzes Masked Diffusion Language Models (MDLMs) along two degrees of freedom—parallelism and generation order—using Average Finalization Parallelism (AFP) and Kendall’s $\tau$, across 58 benchmarks and eight 100B-scale models. It demonstrates that current MDLMs, despite potential for parallelism, lag autoregressive models due to a fundamental parallelization-induced dependency loss, quantified via a conditional total correlation bound that grows with block size. The authors show adaptive decoding dynamics across domains, with higher parallelism correlating with correct outputs and domain-dependent order patterns, and they reveal non-monotonic behaviors such as order disruptions at semantic pivots and improvements on Sudoku-like tasks that leverage non-sequential reasoning. The work proposes a Generate-then-Edit paradigm to mitigate dependency loss while preserving parallel decoding efficiency, providing both theoretical proofs and practical insights for bridging the accuracy gap. Overall, the study offers a comprehensive framework for understanding and unlocking the latent potential of MDLMs in non-linear, efficient language generation and reasoning.

Abstract

Masked Diffusion Language Models (MDLMs) promise parallel token generation and arbitrary-order decoding, yet it remains unclear to what extent current models truly realize these capabilities. We characterize MDLM behavior along two dimensions -- parallelism strength and generation order -- using Average Finalization Parallelism (AFP) and Kendall's tau. We evaluate eight mainstream MDLMs (up to 100B parameters) on 58 benchmarks spanning knowledge, reasoning, and programming. The results show that MDLMs still lag behind comparably sized autoregressive models, mainly because parallel probabilistic modeling weakens inter-token dependencies. Meanwhile, MDLMs exhibit adaptive decoding behavior: their parallelism and generation order vary significantly with the task domain, the stage of reasoning, and whether the output is correct. On tasks that require "backward information" (e.g., Sudoku), MDLMs adopt a solution order that tends to fill easier Sudoku blanks first, highlighting their advantages. Finally, we provide theoretical motivation and design insights supporting a Generate-then-Edit paradigm, which mitigates dependency loss while retaining the efficiency of parallel decoding.

Figures (27)

• Figure 1: Overall performance comparison between AR and DLM models across six evaluation dimensions.
• Figure 2: Comparison at the same parameter scale. Smaller block size $B$ yields superior performance.
• Figure 3: Comparison of intra-chunk parallelism and sequential ordering patterns across different task domains. Repeating samples are analyzed separately because, despite their limited count, each fills the entire 32k context window and disproportionately impacts overall statistical values.
• Figure 4: Comparison of inter-chunk parallelism and sequential ordering patterns across different task domains. Refer to Appendix \ref{['app:tau_block']} for more groupings of $\tau$, and Appendix \ref{['app:fap_block']} for parallel AFP.
• Figure 5: Word clouds of the most frequent parallel decoding combinations across six benchmarks. High-frequency co-occurrences are dominated by fixed phrases and specific special token combinations.

Theorems & Definitions (9)

• Definition 1: Conditional Total Correlation
• Lemma A.1: Factorization Gap Decomposition
• proof
• Theorem A.2: Geometric Mixing of Thresholded Editing
• proof : Proof sketch
• Corollary A.3: Exact recovery under strong block-conditional realizability
• proof : Proof sketch
• Theorem A.4: No-slowdown condition
• proof