Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heavy-Tailed Diffusion Models

Kushagra Pandey, Jaideep Pathak, Yilun Xu, Stephan Mandt, Michael Pritchard, Arash Vahdat, Morteza Mardani

TL;DR

This work repurposes the diffusion framework for heavy-tail estimation using multivariate Student-t distributions, develops a tailored perturbation kernel and derives the denoising posterior based on the conditional Student-t distribution for the backward process, and derives a training objective for heavy-tailed denoisers.

Abstract

Diffusion models achieve state-of-the-art generation quality across many applications, but their ability to capture rare or extreme events in heavy-tailed distributions remains unclear. In this work, we show that traditional diffusion and flow-matching models with standard Gaussian priors fail to capture heavy-tailed behavior. We address this by repurposing the diffusion framework for heavy-tail estimation using multivariate Student-t distributions. We develop a tailored perturbation kernel and derive the denoising posterior based on the conditional Student-t distribution for the backward process. Inspired by $γ$-divergence for heavy-tailed distributions, we derive a training objective for heavy-tailed denoisers. The resulting framework introduces controllable tail generation using only a single scalar hyperparameter, making it easily tunable for diverse real-world distributions. As specific instantiations of our framework, we introduce t-EDM and t-Flow, extensions of existing diffusion and flow models that employ a Student-t prior. Remarkably, our approach is readily compatible with standard Gaussian diffusion models and requires only minimal code changes. Empirically, we show that our t-EDM and t-Flow outperform standard diffusion models in heavy-tail estimation on high-resolution weather datasets in which generating rare and extreme events is crucial.

Heavy-Tailed Diffusion Models

TL;DR

This work repurposes the diffusion framework for heavy-tail estimation using multivariate Student-t distributions, develops a tailored perturbation kernel and derives the denoising posterior based on the conditional Student-t distribution for the backward process, and derives a training objective for heavy-tailed denoisers.

Abstract

Diffusion models achieve state-of-the-art generation quality across many applications, but their ability to capture rare or extreme events in heavy-tailed distributions remains unclear. In this work, we show that traditional diffusion and flow-matching models with standard Gaussian priors fail to capture heavy-tailed behavior. We address this by repurposing the diffusion framework for heavy-tail estimation using multivariate Student-t distributions. We develop a tailored perturbation kernel and derive the denoising posterior based on the conditional Student-t distribution for the backward process. Inspired by -divergence for heavy-tailed distributions, we derive a training objective for heavy-tailed denoisers. The resulting framework introduces controllable tail generation using only a single scalar hyperparameter, making it easily tunable for diverse real-world distributions. As specific instantiations of our framework, we introduce t-EDM and t-Flow, extensions of existing diffusion and flow models that employ a Student-t prior. Remarkably, our approach is readily compatible with standard Gaussian diffusion models and requires only minimal code changes. Empirically, we show that our t-EDM and t-Flow outperform standard diffusion models in heavy-tail estimation on high-resolution weather datasets in which generating rare and extreme events is crucial.

Paper Structure

This paper contains 53 sections, 4 theorems, 148 equations, 12 figures, 9 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion Models
  4. Student-t Distributions
  5. Heavy-Tailed Diffusion Models
  6. Noising Process Design with Student-t Distributions.
  7. Parameterization of the Reverse Posterior
  8. Training with Power Divergences
  9. Sampling
  10. Specific Instantiations: t-EDM
  11. Experiments
  12. Unconditional Generation
  13. Conditional Generation
  14. Discussion and Theoretical Insights
  15. Proofs
  16. ...and 38 more sections

Key Result

Proposition 1

For arbitrary distributions $q$ and $p$, in the limit of $\gamma \rightarrow 0$, $D_\gamma\!\left(q ~ \| ~ p\right)$ converges to $D_{\mathrm{KL}}\!\left(q ~ \| ~ p\right)$. Consequently, for a finite-dimensional dataset with ${\mathbf{x}}_0 \in {\mathbb{R}}^d$ and $\gamma=-\frac{2}{\nu + d}$, under

Figures (12)

  • Figure 1: Toy Illustration. Our proposed diffusion model (t-Diffusion) captures heavy-tailed behavior more accurately than standard Gaussian diffusion, as shown in the histogram comparisons (top panel, x-axis). The framework allows for controllable tail estimation using a hyperparameter $\nu$, which can be adjusted for each dimension. Lower $\nu$ values model heavier tails, while higher values approach Gaussian diffusion. (Best viewed when zoomed in; see App. \ref{['app:exp_toy']} for details)
  • Figure 2: Training (t-EDM)
  • Figure 3: Sample 1-d histogram comparison between EDM and t-EDM on the test set for the Vertically Integrated Liquid (VIL) channel. t-EDM captures heavy-tailed behavior more accurately than other baselines. INC: Inverse CDF Normalization, PCP: Per-Channel Preconditioning
  • Figure 4: Variation of the mean and standard deviation of the ratio $(\nu + d_1)/(\nu + d)$ with $\nu$ across diffusion sampling trajectory for the toy dataset. As $\nu$ decreases, the mean ratio and its standard deviation increase, leading to large score multiplier weights.
  • Figure 5: Training (t-Flow)
  • ...and 7 more figures

Theorems & Definitions (7)

  • Proposition 1
  • Proposition 2
  • proof
  • Proposition 1
  • proof
  • Proposition 2
  • proof