Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Discrete Diffusion Models

Dario Shariatian, Alain Durmus, Stefano Peluchetti

TL;DR

Latent Discrete Diffusion Models (LDDMs) address the brittleness of masked discrete diffusion by introducing a continuous latent channel that carries cross-token dependencies. Two instantiations are proposed: FUJI-LDDMs, which jointly denoise tokens and latents, and SEQ-LDDMs, which first denoise the latent stream and then condition the discrete chain on it. The authors derive ELBO-based training objectives for both variants and demonstrate improved unconditional generation, especially at low sampling budgets, on synthetic and LM1B language modeling tasks. The latent channel can be learned jointly or via a frozen encoder, producing gains in coherence and generation quality while preserving token-level parallelism. Overall, LDDMs offer a principled way to inject soft, joint structure into discrete diffusion without sacrificing the efficiency of parallel token updates.

Abstract

We study discrete diffusion for language and other categorical data and focus on a common limitation of masked denoisers: reverse transitions typically factorize across positions, which can weaken joint structure and degrade quality in few-step generation. We propose \emph{Latent Discrete Diffusion Models} (LDDMs), which couple a masked discrete diffusion over tokens with a continuous diffusion over latent embeddings. The latent channel provides a softer signal and carries cross-token dependencies that help resolve ambiguities. We present two instantiations: (i) FUJI-LDDMs, which perform fully joint denoising of tokens and latents, and (ii) SEQ-LDDMs, which sequentially resolve the latent and then the discrete chain conditionally on it. For both variants we derive ELBO-style objectives and discuss design choices to learn informative latents yet amenable to diffusoin modeling. In experiments, LDDMs yield improvements on unconditional generation metrics as compared to state-of-the-art masked discrete diffusion baselines, and are effective at lower sampling budgets, where unmasking many tokens per step is desirable.

Latent Discrete Diffusion Models

TL;DR

Latent Discrete Diffusion Models (LDDMs) address the brittleness of masked discrete diffusion by introducing a continuous latent channel that carries cross-token dependencies. Two instantiations are proposed: FUJI-LDDMs, which jointly denoise tokens and latents, and SEQ-LDDMs, which first denoise the latent stream and then condition the discrete chain on it. The authors derive ELBO-based training objectives for both variants and demonstrate improved unconditional generation, especially at low sampling budgets, on synthetic and LM1B language modeling tasks. The latent channel can be learned jointly or via a frozen encoder, producing gains in coherence and generation quality while preserving token-level parallelism. Overall, LDDMs offer a principled way to inject soft, joint structure into discrete diffusion without sacrificing the efficiency of parallel token updates.

Abstract

We study discrete diffusion for language and other categorical data and focus on a common limitation of masked denoisers: reverse transitions typically factorize across positions, which can weaken joint structure and degrade quality in few-step generation. We propose \emph{Latent Discrete Diffusion Models} (LDDMs), which couple a masked discrete diffusion over tokens with a continuous diffusion over latent embeddings. The latent channel provides a softer signal and carries cross-token dependencies that help resolve ambiguities. We present two instantiations: (i) FUJI-LDDMs, which perform fully joint denoising of tokens and latents, and (ii) SEQ-LDDMs, which sequentially resolve the latent and then the discrete chain conditionally on it. For both variants we derive ELBO-style objectives and discuss design choices to learn informative latents yet amenable to diffusoin modeling. In experiments, LDDMs yield improvements on unconditional generation metrics as compared to state-of-the-art masked discrete diffusion baselines, and are effective at lower sampling budgets, where unmasking many tokens per step is desirable.
Paper Structure (66 sections, 3 theorems, 65 equations, 4 figures, 3 tables)

This paper contains 66 sections, 3 theorems, 65 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Our approach.
  3. Contributions.
  4. Preliminaries on Diffusion Models
  5. Notation.
  6. Continuous Diffusion Models
  7. Masked Discrete Diffusion Models
  8. Forward process.
  9. Generative process.
  10. Objective function.
  11. Limits and Motivation
  12. Latent Discrete Diffusion Models: LDDMs
  13. General Framework for LDDMs
  14. Forward process.
  15. FUJI-LDDM: FUlly JoInt denoising.
  16. ...and 51 more sections

Key Result

Proposition 1

liu2025discretecopuladiffusion Let $q^{\mathsf{X}}$ be the path measure of the forward masked diffusion process on $\mathsf{X}^S$. Let $p^{\mathsf{X}, \theta}$ be any factorized Markovian reverse model initialized at the all-mask state. Then where $\mathrm{L}^{\mathsf{X}}$ is the negative ELBO in eq:mdlm_discrete_nelbo, $\mathrm{H}$ is the Shannon entropy, and $q^{\mathsf{X}, i}_{t-1\mid t}$ deno

Figures (4)

  • Figure 1: Two-channel schematic of FUJI-LDDMs. Dashed blue arrows show the forward noising process, and green/red lines show the generative joint diffusion process.
  • Figure 2: SWD$\downarrow$ as a function of function evaluation on the data channel. Latent steps for SEQ are fixed to $T_Y=64$.
  • Figure 3: Values of $q^{\mathsf{X}, i}_{0|y}(1 | y)$ across coordinates for $y \in [0, 1]$.
  • Figure 4: Classical choices for noise schedules. For the cosine schedule, we set $s = 0.008$.

Theorems & Definitions (4)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • proof : Proof of \ref{['prop:latent_factorization_limits']}