Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models

Etrit Haxholli, Yeti Z. Gurbuz, Ogul Can, Eli Waxman

TL;DR

This work tackles the challenge of modeling discrete language with diffusion, introducing a CTMC-based discrete diffusion framework and proving three KL-divergence theorems that yield a tighter perplexity bound via $J_2$. It replaces ratio-matching training with cross-entropy discrete diffusion (CEDD) and introduces roulette diffusion, featuring an analytic matrix exponential to enable efficient training and refinement during unmasking. Empirically, CEDD and its variants consistently outperform SEDD across absorb, uniform, and roulette dynamics, delivering up to ~10% lower perplexity and ~15% faster training, with additional gains in spelling correction tasks. The results advance practical, scalable discrete diffusion models for language with improved evaluation bounds and generation efficiency.

Abstract

While continuous diffusion models excel in modeling continuous distributions, their application to categorical data has been less effective. Recent work has shown that ratio-matching through score-entropy within a continuous-time discrete Markov chain (CTMC) framework serves as a competitive alternative to autoregressive models in language modeling. To enhance this framework, we first introduce three new theorems concerning the KL divergence between the data and learned distribution. Our results serve as the discrete counterpart to those established for continuous diffusion models and allow us to derive an improved upper bound of the perplexity. Second, we empirically show that ratio-matching performed by minimizing the denoising cross-entropy between the clean and corrupted data enables models to outperform those utilizing score-entropy with up to 10% lower perplexity/generative-perplexity, and 15% faster training steps. To further support our findings, we introduce and evaluate a novel CTMC transition-rate matrix that allows prediction refinement, and derive the analytic expression for its matrix exponential which facilitates the computation of conditional ratios thus enabling efficient training and generation.

Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models

TL;DR

This work tackles the challenge of modeling discrete language with diffusion, introducing a CTMC-based discrete diffusion framework and proving three KL-divergence theorems that yield a tighter perplexity bound via . It replaces ratio-matching training with cross-entropy discrete diffusion (CEDD) and introduces roulette diffusion, featuring an analytic matrix exponential to enable efficient training and refinement during unmasking. Empirically, CEDD and its variants consistently outperform SEDD across absorb, uniform, and roulette dynamics, delivering up to ~10% lower perplexity and ~15% faster training, with additional gains in spelling correction tasks. The results advance practical, scalable discrete diffusion models for language with improved evaluation bounds and generation efficiency.

Abstract

While continuous diffusion models excel in modeling continuous distributions, their application to categorical data has been less effective. Recent work has shown that ratio-matching through score-entropy within a continuous-time discrete Markov chain (CTMC) framework serves as a competitive alternative to autoregressive models in language modeling. To enhance this framework, we first introduce three new theorems concerning the KL divergence between the data and learned distribution. Our results serve as the discrete counterpart to those established for continuous diffusion models and allow us to derive an improved upper bound of the perplexity. Second, we empirically show that ratio-matching performed by minimizing the denoising cross-entropy between the clean and corrupted data enables models to outperform those utilizing score-entropy with up to 10% lower perplexity/generative-perplexity, and 15% faster training steps. To further support our findings, we introduce and evaluate a novel CTMC transition-rate matrix that allows prediction refinement, and derive the analytic expression for its matrix exponential which facilitates the computation of conditional ratios thus enabling efficient training and generation.

Paper Structure

This paper contains 45 sections, 8 theorems, 199 equations, 8 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Notation
  3. Markov Chains Over Finite-State Spaces
  4. SEDD: Estimating the Ratios via Score Entropy
  5. Discrete Diffusion for Language Modeling - Token Level Transitions
  6. Methodology and Theoretical Results
  7. Cross Entropy and KL Divergence Results
  8. Roulette Discrete Diffusion
  9. Efficient Estimation of $J_2$
  10. Modeling Ratios via CEDD
  11. Experiments
  12. Generative Perplexity Comparisons
  13. Perplexity Comparisons
  14. Comparing the Two Upper Bounds
  15. Spelling Correction
  16. ...and 30 more sections

Key Result

Theorem 1

Define a CTMC with transition matrix ${{\bm{Q}}}_{t}$ that runs from time $0$ to $1$. The true reverse process defines a probability evolution $p_t$ from $p_1$ to the data distribution $p_0$, while the learned reverse process induces the evolution $p^{\theta}_t$ from the reference distribution $p^{\ where $\ell(a, b) = \left( b-a \log b + K (a) \right)$ and $K(a) = a(\log a - 1)$.

Figures (8)

  • Figure 1: Scaling of Generative Perplexity vs sampling steps for SEDDs (loaded) and CEDD* absorb.
  • Figure 2: Filtered samples from SEDDs and CEDD* absorb, L=1024. The conditional part is highlighted in bold.
  • Figure 3: Upper bounds $J_1$ and $J_2$ of CEDD* absorb L=1024 for different testing sets.
  • Figure 4: The conditional ratios at position $i$ over the vocabulary in the two cases. The square represents the absorb state. The value of $a$ is $1-e^{-\sigma_t}$.
  • Figure 5: The conditional ratios at position $i$ over the vocabulary in the two cases. We define $b=\frac{1}{n=V}(1-e^{-\sigma_t})$ and $c=1-(n-1)b$.
  • ...and 3 more figures

Theorems & Definitions (14)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Proposition 5
  • Proposition 6
  • Lemma 1
  • proof
  • proof
  • proof
  • ...and 4 more