Information Theoretic Learning for Diffusion Models with Warm Start

TL;DR

This work extends likelihood estimation for diffusion models by deriving a generalized KL–Fisher relationship that holds under arbitrary isotropic noise, enabling a variance-aware, warm-start diffusion framework. It introduces a tractable upper bound on the mismatched entropy via thermodynamic diffusion, a pointwise likelihood bound, and efficient variance-reduction techniques (importance sampling and warm-up dequantization) that stabilize training. The approach achieves state-of-the-art or competitive negative log-likelihoods (in bits/dim) on CIFAR-10 and ImageNet variants without data augmentation, and provides a practical path to density estimation and potentially lossless compression with diffusion models. It also broadens applicability to discrete data through training-free dequantization and clarifies the role of noise schedules in likelihood optimization, while noting limitations in constructing alternative-noise generative processes and the need for further hyperparameter exploration.

Abstract

Generative models that maximize model likelihood have gained traction in many practical settings. Among them, perturbation based approaches underpin many strong likelihood estimation models, yet they often face slow convergence and limited theoretical understanding. In this paper, we derive a tighter likelihood bound for noise driven models to improve both the accuracy and efficiency of maximum likelihood learning. Our key insight extends the classical KL divergence Fisher information relationship to arbitrary noise perturbations, going beyond the Gaussian assumption and enabling structured noise distributions. This formulation allows flexible use of randomized noise distributions that naturally account for sensor artifacts, quantization effects, and data distribution smoothing, while remaining compatible with standard diffusion training. Treating the diffusion process as a Gaussian channel, we further express the mismatched entropy between data and model, showing that the proposed objective upper bounds the negative log-likelihood (NLL). In experiments, our models achieve competitive NLL on CIFAR-10 and SOTA results on ImageNet across multiple resolutions, all without data augmentation, and the framework extends naturally to discrete data.

Figures (6)

• Figure 1: Toy example illustrating our method (left) and samples generated on CIFAR-10 (right). We apply an identical warm-up channel to both the data $p(\mathbf{x})$ and the model distribution $q(\mathbf{x};\boldsymbol{\theta})$: injecting arbitrary noise $\boldsymbol{\Psi}$ produces smoothed data $\tilde{\mathbf{x}}$ and a correspondingly perturbed model $\tilde{q}_\theta$. This results in two variance-regime mismatched channels, a low-variance arbitrary-perturbation channel and a high-variance Gaussian noise regime, that share identical channel dynamics but differ in their priors, under which training aligns the Gaussian-perturbed distributions and learns the denoising map.
• Figure 2: Random samples from our model trained on CIFAR-10 for 300000 parameter updates with EMA. The model was trained in VDM kingma2021variational endpoints, and sampled using 1000 sampling timestep.
• Figure 3: Random samples from our model trained on CIFAR-10 for 300000 parameter updates with EMA. The model was trained with our endpoints, and sampled using 1000 sampling timestep.
• Figure 4: Random samples from our model trained on ImageNet32 for 300000 parameter updates. The model was trained in VDM kingma2021variational endpoints, and sampled using 1000 sampling timestep.
• Figure 5: Random samples from our model trained on ImageNet32 for 300000 parameter updates. The model was trained with our endpoints, and sampled using 1000 sampling timestep.

Theorems & Definitions (24)

• Theorem 1: Score Matching as the Small-Noise Limit of KL Divergence
• proof
• Proposition 1: Thermodynamic Decomposition of Mismatched Entropy
• proof
• Theorem 2
• proof
• Proposition 2: de Bruijn identity with arbitrary noise rioul2010information
• Definition 1
• Definition 2
• Definition 3