CANDI: Hybrid Discrete-Continuous Diffusion Models

TL;DR

This work proposes CANDI (Continuous ANd DIscrete diffusion), a hybrid framework that decouples discrete and continuous corruption, enabling simultaneous learning of both conditional structure and continuous geometry.

Abstract

While continuous diffusion has shown remarkable success in continuous domains such as image generation, its direct application to discrete data has underperformed compared to purely discrete formulations. This gap is counterintuitive, given that continuous diffusion learns score functions that enable joint evolution across multiple positions. To understand this gap, we introduce token identifiability as an analytical framework for understanding how Gaussian noise corrupts discrete data through two mechanisms: discrete identity corruption and continuous rank degradation. We reveal that these mechanisms scale differently with vocabulary size, creating a temporal dissonance: at noise levels where discrete corruption preserves enough structure for conditional learning, continuous denoising is trivial; at noise levels where continuous denoising is meaningful, discrete corruption destroys nearly all conditional structure. To solve this, we propose CANDI (Continuous ANd DIscrete diffusion), a hybrid framework that decouples discrete and continuous corruption, enabling simultaneous learning of both conditional structure and continuous geometry. We empirically validate the temporal dissonance phenomenon and demonstrate that CANDI successfully avoids it. This unlocks the benefits of continuous diffusion for discrete spaces: on controlled generation, CANDI enables classifier-based guidance with off-the-shelf classifiers through simple gradient addition; on text generation, CANDI outperforms masked diffusion at low NFE, demonstrating the value of learning continuous gradients for discrete spaces. We include the code on the project page available here: https://patrickpynadath1.github.io/candi-lander

Figures (20)

• Figure 1: We provide a visual comparison of discrete, continuous, and hybrid diffusion. Discrete diffusion leverages discrete conditional structure, but lacks the ability to perform joint updates due to sampling each position independently. Continuous diffusion performs joint updates, but lacks discrete conditional structure. Our proposed hybrid diffusion, CANDI, can leverage both discrete conditional structure and joint updates.
• Figure 2: We visualize the two forms of corruption for token identifiability and visualize the asymmetric scaling with the number of categories $|V|$. Token identifiability, i.e., how the correct token relates to the noisy latent, is affected by Gaussian noise in two distinct mechanisms: (1) discrete corruption, determining whether the correct token can be obtained by taking the argmax of the noisy latent, and (2) continuous corruption, measuring how much signal from the correct token has degraded. While these two are different facets of the same underlying signal, they scale very differently with increases in vocabulary size $|V|$.
• Figure 3: We plot $\rho(\sigma)$ and $r(\sigma)$ for different numbers of categories to illustrate the divergence between discrete identity corruption and continuous rank degradation. At small $|V|$, both mechanisms change at roughly the same rate through the forward process, enabling the model to learn both conditional structure and continuous denoising simultaneously. At larger values of $|V|$, even when half of the positions are argmax corrupted, the correct token index remain larger than 99% of the other coordinates. Thus when the model can learn conditional structure, it does not learn the continuous gradient; and when it learns the continuously denoise, it does not learn conditional structure.
• Figure 4: We demonstrate that our hybrid kernel eliminates the temporal dissonance induced by a large number of categories by decoupling discrete and continuous corruption. While the Gaussian kernel (blue lines) requires complete discrete corruption $\rho=1$ to achieve meaningful continuous degradation at large vocabulary size, our hybrid approach maintains a linear relationship between $\rho$ and $r$ by explicitly controlling discrete corruption through a masking schedule.
• Figure 5: We demonstrate that single-point evaluations for perplexity and entropy can lead to misleading results. For each plot, we adjust the temperature of each method ($\tau \in [.875, 1.0]$) to achieve reasonable ranges of entropy (5.2-5.7, annotated along the curves) sahoo2025diffusionzheng2024masked. We show that small changes in this hyperparameter can produce entirely different relative performance rankings. We provide details on temperature settings for each plot in Appendix \ref{['app:sec_6:tuned-temperature-details']}.