Diffusion Models with Heavy-Tailed Targets: Score Estimation and Sampling Guarantees
Yifeng Yu, Lu Yu
TL;DR
This work extends score-based diffusion theory to heavy-tailed target distributions by analyzing a Gaussian forward diffusion with targets in a Sobolev class $\mathcal{P}_{\mathcal{S}}(\beta,L)$ and tail index $\gamma$. It introduces a kernel density-based score estimator with a thresholded form and proves sharp minimax rates for score estimation under both polynomial and exponential tails, revealing a qualitative dichotomy: polynomial tails incur a tail-parameter dependent rate, while exponential tails yield near-parametric rates. The authors also derive sampling guarantees for the continuous reverse diffusion in total variation, showing minimax-optimal rates $n^{-{\frac{\beta}{2\beta+d}}}$ under exponential tails and a $\gamma$-dependent rate under polynomial tails, with an open question on optimality of the latter. Taken together, these results establish the statistical limits of score estimation and diffusion-based sampling for heavy-tailed targets, broadening the theoretical understanding beyond light-tailed settings and guiding practical algorithm design under tail heaviness.
Abstract
Score-based diffusion models have become a powerful framework for generative modeling, with score estimation as a central statistical bottleneck. Existing guarantees for score estimation largely focus on light-tailed targets or rely on restrictive assumptions such as compact support, which are often violated by heavy-tailed data in practice. In this work, we study conventional (Gaussian) score-based diffusion models when the target distribution is heavy-tailed and belongs to a Sobolev class with smoothness parameter $β>0$. We consider both exponential and polynomial tail decay, indexed by a tail parameter $γ$. Using kernel density estimation, we derive sharp minimax rates for score estimation, revealing a qualitative dichotomy: under exponential tails, the rate matches the light-tailed case up to polylogarithmic factors, whereas under polynomial tails the rate depends explicitly on $γ$. We further provide sampling guarantees for the associated continuous reverse dynamics. In total variation, the generated distribution converges at the minimax optimal rate $n^{-β/(2β+d)}$ under exponential tails (up to logarithmic factors), and at a $γ$-dependent rate under polynomial tails. Whether the latter sampling rate is minimax optimal remains an open question. These results characterize the statistical limits of score estimation and the resulting sampling accuracy for heavy-tailed targets, extending diffusion theory beyond the light-tailed setting.