Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models
Song Mei, Yuchen Wu
TL;DR
This work reframes diffusion-based generative modeling for high-dimensional graphical data by linking score-function learning to variational inference denoisers via Tweedie’s formula, enabling sample-efficient learning when the data come from graphical models such as Ising and sparse-coding systems. By proving a modular score-estimation bound under variational-approximation assumptions and combining it with established DDPM discretization analyses, the authors derive finite-sample KL guarantees for diffusion-based sampling with neural-network score estimators. They validate the approach across Ising, latent-variable Ising, conditional Ising, and sparse coding models, highlighting regimes where variational-free-energy corrections (e.g., TAP) ensure consistent denoisers. The framework offers a pathway to scalable diffusion samplers on complex discrete structures and suggests broad applicability beyond DDPM to stochastic-localization methods, with potential for improved architectures via algorithm unrolling. The results illuminate when neural-score learning is tractable in high dimensions and outline directions to relax assumptions and broaden model classes.
Abstract
We investigate the approximation efficiency of score functions by deep neural networks in diffusion-based generative modeling. While existing approximation theories utilize the smoothness of score functions, they suffer from the curse of dimensionality for intrinsically high-dimensional data. This limitation is pronounced in graphical models such as Markov random fields, common for image distributions, where the approximation efficiency of score functions remains unestablished. To address this, we observe score functions can often be well-approximated in graphical models through variational inference denoising algorithms. Furthermore, these algorithms are amenable to efficient neural network representation. We demonstrate this in examples of graphical models, including Ising models, conditional Ising models, restricted Boltzmann machines, and sparse encoding models. Combined with off-the-shelf discretization error bounds for diffusion-based sampling, we provide an efficient sample complexity bound for diffusion-based generative modeling when the score function is learned by deep neural networks.