Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simple Denoising Diffusion Language Models

Huaisheng Zhu, Zhengyu Chen, Shijie Zhou, Zhihui Xie, Yige Yuan, Zhimeng Guo, Siyuan Xu, Hangfan Zhang, Vasant Honavar, Teng Xiao

TL;DR

The paper tackles performance gaps in discrete diffusion language models, particularly in few-step generation, by proposing Simple Denoising Diffusion Language Models (SDDLM) that simplify USDM losses. By training only on noise-replaced tokens and viewing denoising as self-supervised learning, the approach stabilizes training and matches ELBO-level performance, with a contrastive-inspired negative-gradient extension (SDDLM-V1/V2) that further improves generation quality. Empirical results on LM1B and OpenWebText show that SDDLM maintains competitive perplexity while delivering higher-quality samples, especially when negative gradients are used, though ELBO-based perplexity trends can diverge from sampling quality. The method offers a scalable, efficient pathway for diffusion-based language modeling, potentially enabling faster sampling with fewer steps and easier scaling to larger architectures.

Abstract

Diffusion models have recently been extended to language generation through Masked Diffusion Language Models (MDLMs), which achieve performance competitive with strong autoregressive models. However, MDLMs tend to degrade in the few-step regime and cannot directly adopt existing few-step distillation methods designed for continuous diffusion models, as they lack the intrinsic property of mapping from noise to data. Recent Uniform-state Diffusion Models (USDMs), initialized from a uniform prior, alleviate some limitations but still suffer from complex loss formulations that hinder scalability. In this work, we propose a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training and matching ELBO-level performance. Furthermore, by framing denoising as self-supervised learning, we introduce a simple modification to our denoising loss with contrastive-inspired negative gradients, which is practical and yield additional improvements in generation quality.

Simple Denoising Diffusion Language Models

TL;DR

The paper tackles performance gaps in discrete diffusion language models, particularly in few-step generation, by proposing Simple Denoising Diffusion Language Models (SDDLM) that simplify USDM losses. By training only on noise-replaced tokens and viewing denoising as self-supervised learning, the approach stabilizes training and matches ELBO-level performance, with a contrastive-inspired negative-gradient extension (SDDLM-V1/V2) that further improves generation quality. Empirical results on LM1B and OpenWebText show that SDDLM maintains competitive perplexity while delivering higher-quality samples, especially when negative gradients are used, though ELBO-based perplexity trends can diverge from sampling quality. The method offers a scalable, efficient pathway for diffusion-based language modeling, potentially enabling faster sampling with fewer steps and easier scaling to larger architectures.

Abstract

Diffusion models have recently been extended to language generation through Masked Diffusion Language Models (MDLMs), which achieve performance competitive with strong autoregressive models. However, MDLMs tend to degrade in the few-step regime and cannot directly adopt existing few-step distillation methods designed for continuous diffusion models, as they lack the intrinsic property of mapping from noise to data. Recent Uniform-state Diffusion Models (USDMs), initialized from a uniform prior, alleviate some limitations but still suffer from complex loss formulations that hinder scalability. In this work, we propose a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training and matching ELBO-level performance. Furthermore, by framing denoising as self-supervised learning, we introduce a simple modification to our denoising loss with contrastive-inspired negative gradients, which is practical and yield additional improvements in generation quality.
Paper Structure (13 sections, 8 equations, 3 figures, 2 tables)

This paper contains 13 sections, 8 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Denoising Diffusion Models
  4. Discrete Diffusion Models
  5. Related Works
  6. Denoising Diffusion Language Models
  7. Experiment
  8. Experimental Setup
  9. Sample Quality Comparison
  10. Likelihood Evaluation
  11. Conclusion
  12. Details of Hyperparameter
  13. Generation Examples

Figures (3)

  • Figure 1: Validation Gen PPL of different models over training steps.
  • Figure 2: Validation entropy of different models over training steps.
  • Figure 3: Validation PPL of different models over training steps.