Latent Shadows: The Gaussian-Discrete Duality in Masked Diffusion

TL;DR

This work tackles the inefficiency of discrete masked diffusion for language modeling by deriving a formal Masked Diffusion Duality, proving the masked process is the projection of a continuous Gaussian diffusion with deterministic latent trajectories. It then introduces Scalar Trajectory Locking to reduce high-dimensional latent coupling to a single scalar threshold, enabling Masked Consistency Distillation (MCD) with a Hybrid Consistency Objective that distills trajectories without numerical ODE solvers. Empirically, MCD achieves a 16× speedup while maintaining or improving generation quality over strong baselines, and scales effectively to large backbones on OpenWebText. The theoretical link between masked and continuous diffusion, together with the practical MCD framework, offers a robust foundation for efficient high-quality discrete generation.

Abstract

Masked discrete diffusion is a dominant paradigm for high-quality language modeling where tokens are iteratively corrupted to a mask state, yet its inference efficiency is bottlenecked by the lack of deterministic sampling tools. While diffusion duality enables deterministic distillation for uniform models, these approaches generally underperform masked models and rely on complex integral operators. Conversely, in the masked domain, prior methods typically assume the absence of deterministic trajectories, forcing a reliance on stochastic distillation. To bridge this gap, we establish explicit Masked Diffusion Duality, proving that the masked process arises as the projection of a continuous Gaussian process via a novel maximum-value index preservation mechanism. Furthermore, we introduce Masked Consistency Distillation (MCD), a principled framework that leverages this duality to analytically construct the deterministic coupled trajectories required for consistency distillation, bypassing numerical ODE solvers. This result strictly improves upon prior stochastic distillation methods, achieving a 16$\times$ inference speedup without compromising generation quality. Our findings not only provide a solid theoretical foundation connecting masked and continuous diffusion, but also unlock the full potential of consistency distillation for high-performance discrete generation. Our code is available at https://anonymous.4open.science/r/MCD-70FD.

Figures (3)

• Figure 1: An illustration of the Masked Diffusion Duality. Top: The discrete masking process (blue) directly samples $\mathbf{z}_t$ from $\mathbf{x}_0$. Bottom: The underlying continuous latent Gaussian process (orange). The projection operator $\mathcal{P}$ maps the continuous latents $\mathbf{w}_t \in \mathbb{R}^K$ to the discrete observations $\mathbf{z}_t$ by thresholding the signal strength against the maximum relative noise. This establishes a deterministic link between the marginal distributions of the two processes.
• Figure 2: Visualization of Coupled Trajectory Construction and Hybrid Objective. By sharing latent noise $u$, we enable three distinct supervision regimes: (1) Identity (Gray, left) where both see the input, requiring no gradient update; (2) Distillation (Orange, middle) where both views are masked ($u > \gamma_s$), ensuring consistency on uncertain regions; (3) Reconstruction (Green, right) where the teacher reveals tokens masked for the student ($\gamma_t < u \le \gamma_s$), providing hard ground-truth supervision.
• Figure 3: Performance across Sampling Steps. Generative Perplexity (Gen PPL) for the Teacher model (MDLM, Grey-Blue) and successive rounds of Masked Consistency Distillation (MCD, Orange). By zooming in on the PPL $\le 200$ regime, we highlight that MCD matches the teacher's performance with significant speedup.

Theorems & Definitions (6)

• Lemma 3.1: SNR Calibration
• proof
• Proposition 3.2: Deterministic Discrete Trajectories
• proof
• Theorem 3.3: Scalar Trajectory Locking
• proof