Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Role of Discreteness in Diffusion LLMs

Ziqi Jin, Bin Wang, Xiang Lin, Lidong Bing, Aixin Sun

TL;DR

This work interrogates diffusion-based language models by separating diffusion mechanics from language-specific constraints via five guiding properties, then classifying existing models into continuous and discrete families. It reveals two central gaps: diffusion’s uniform corruption can squander information that is unevenly distributed across positions, and token-wise marginal denoising fails to enforce multi-token dependencies during parallel decoding. Through theoretical framing and empirical analyses, the paper demonstrates how current DLMs only partially satisfy the proposed properties and suggests directions—such as semantically aware corruption or sequence-level objectives—to achieve more coherent diffusion-based language models. The insights offer a roadmap for designing diffusion processes that better respect text structure, with potential benefits in editing flexibility, computation-length tradeoffs, and data efficiency for language tasks.

Abstract

Diffusion models offer appealing properties for language generation, such as parallel decoding and iterative refinement, but the discrete and highly structured nature of text challenges the direct application of diffusion principles. In this paper, we revisit diffusion language modeling from the view of diffusion process and language modeling, and outline five properties that separate diffusion mechanics from language-specific requirements. We first categorize existing approaches into continuous diffusion in embedding space and discrete diffusion over tokens. We then show that each satisfies only part of the five essential properties and therefore reflects a structural trade-off. Through analyses of recent large diffusion language models, we identify two central issues: (i) uniform corruption does not respect how information is distributed across positions, and (ii) token-wise marginal training cannot capture multi-token dependencies during parallel decoding. These observations motivate diffusion processes that align more closely with the structure of text, and encourage future work toward more coherent diffusion language models.

On the Role of Discreteness in Diffusion LLMs

TL;DR

This work interrogates diffusion-based language models by separating diffusion mechanics from language-specific constraints via five guiding properties, then classifying existing models into continuous and discrete families. It reveals two central gaps: diffusion’s uniform corruption can squander information that is unevenly distributed across positions, and token-wise marginal denoising fails to enforce multi-token dependencies during parallel decoding. Through theoretical framing and empirical analyses, the paper demonstrates how current DLMs only partially satisfy the proposed properties and suggests directions—such as semantically aware corruption or sequence-level objectives—to achieve more coherent diffusion-based language models. The insights offer a roadmap for designing diffusion processes that better respect text structure, with potential benefits in editing flexibility, computation-length tradeoffs, and data efficiency for language tasks.

Abstract

Diffusion models offer appealing properties for language generation, such as parallel decoding and iterative refinement, but the discrete and highly structured nature of text challenges the direct application of diffusion principles. In this paper, we revisit diffusion language modeling from the view of diffusion process and language modeling, and outline five properties that separate diffusion mechanics from language-specific requirements. We first categorize existing approaches into continuous diffusion in embedding space and discrete diffusion over tokens. We then show that each satisfies only part of the five essential properties and therefore reflects a structural trade-off. Through analyses of recent large diffusion language models, we identify two central issues: (i) uniform corruption does not respect how information is distributed across positions, and (ii) token-wise marginal training cannot capture multi-token dependencies during parallel decoding. These observations motivate diffusion processes that align more closely with the structure of text, and encourage future work toward more coherent diffusion language models.
Paper Structure (22 sections, 2 equations, 3 figures, 1 table)

This paper contains 22 sections, 2 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Continuous DLMs
  4. Discrete DLMs
  5. Advantages of DLMs
  6. Flexible Editing.
  7. Token generation is not one-to-one with forward operations.
  8. Multi-View Data Modeling.
  9. Properties for Diffusion LLMs
  10. How Current DLMs Fit the Properties
  11. Trade-offs.
  12. Core Challenges for DLMs
  13. Smooth Corruption does not mean Even Information Loss
  14. Empirical Study.
  15. Future Direction.
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Core Advantages of Diffusion Language Models (DLMs) over Autoregressive (AR) Models.
  • Figure 2: An example from LIMA dataset zhou2023lima illustrating the probability distribution analysis of top-3 tokens by LLaDA-Instruct with 128 masked positions when prompt with "Can brain cells move? By movement I mean long distance migration (preferably within the brain only).". We found early [MASK] are more determistic, while distant ones collapse to frequency-dominant. The Experiment Setting is detailed in Appendix \ref{['sec:exp_detail']}.
  • Figure 3: The " Marginal Trap": A toy example shows that the model learns from "He likes apple" and "I play tennis" (50% each). However, parallel decoding samples them independently resulting each position to be 50% for both samples at each tokon position. Directly sampling form these distribution may create a path (I $\to$ likes $\to$ tennis) that never existed in the training data.