Deterministic Discrete Denoising
Hideyuki Suzuki, Hiroshi Yamashita
TL;DR
This work tackles the challenge of derandomizing discrete-state diffusion by introducing a time-dependent herding-based reverse process that deterministically maps noise to data while preserving probability mass. The method serves as a drop-in alternative to stochastic reverse steps, with randomness confined to initialization and theoretical guarantees of $O(1/T)$ convergence in empirical feature distributions. Across character- and word-level text and categorical image generation, the approach yields consistent gains in sample quality and efficiency, often outperforming stochastic baselines with substantially fewer denoising steps. Overall, the results demonstrate that deterministic reverse processes can be effective in discrete diffusion, opening pathways to continuous-time variants and hardware-friendly implementations.
Abstract
We propose a deterministic denoising algorithm for discrete-state diffusion models based on Markov chains. The generative reverse process is derandomized by introducing a variant of the herding algorithm with weakly chaotic dynamics, which induces deterministic discrete state transitions. Our approach is a direct replacement for the stochastic denoising process, requiring neither retraining nor continuous state embeddings. We demonstrate consistent improvements in both efficiency and sample quality on text and image generation tasks. Thus, this simple derandomization approach is expected to enhance the significance of discrete diffusion in generative modeling. Furthermore, our results reveal that deterministic reverse processes, well established in continuous diffusion, can also be effective in discrete state spaces.