Why mask diffusion does not work

TL;DR

The paper analyzes mask diffusion language models and shows that their outputs are effectively marginal distributions, which hinders true parallel generation and bidirectional attention. It provides a theoretical framework and empirical evidence demonstrating distance-dependent marginals, lack of joint coherence, and near-autoregressive generation behavior, even under masking. The authors propose an inference/training strategy—semi-autoregressive generation in small blocks with blockwise reverse-order training—that better aligns with current mask-diffusion behavior, though it does not fundamentally enable genuine parallelism. The work highlights the need for diffusion approaches that support true parallel generation and effective bidirectional context, while offering practical guidance for training and inference under the current paradigm.

Abstract

The main advantages of diffusion language models over autoregressive (AR) models lie in their ability to support parallel generation and bidirectional attention, enabling a more controllable generation process. In recent years, open-source mask diffusion language models have emerged, most of which are based on a variant known as absorbing diffusion. However, this paper demonstrates why mask diffusion faces inherent difficulties in achieving parallel generation and bidirectional attention. We also propose the most effective training and inference strategies for mask diffusion.

Figures (4)

• Figure 1: Model Output as Marginal Distributions
• Figure 2: Initial state during inference
• Figure 3: True average max prob and upper bound estimation
• Figure 4: Semi-AR Generation (Block size=4)

Theorems & Definitions (1)

• Definition