Generative Modeling with Bayesian Sample Inference

TL;DR

Bayesian Sample Inference (BSI) presents a generative framework that treats the unknown data sample ${\bm{x}}$ as the object of iterative Bayesian updates driven by noisy measurements ${\bm{y}}_i$. A predictor $f_{\theta}$ alternates with posterior updates to progressively refine ${\bm{x}}$, linking diffusion models and Bayesian Flow Networks within a unified Bayesian view. The authors derive an evidence lower bound (ELBO) for training and introduce variance-reduction via importance sampling, showing competitive log-likelihoods on CIFAR10 and ImageNet32 along with improved sample quality on ImageNet32. They implement preconditioning and precision-encoding to stabilize learning, experiment on ImageNet32 and CIFAR10 with U-Net and DiT backbones, and discuss how BFNs and diffusion models arise as special cases of the BSI framework, suggesting a broadly applicable probabilistic lens for generative modeling.

Abstract

We derive a novel generative model from iterative Gaussian posterior inference. By treating the generated sample as an unknown variable, we can formulate the sampling process in the language of Bayesian probability. Our model uses a sequence of prediction and posterior update steps to iteratively narrow down the unknown sample starting from a broad initial belief. In addition to a rigorous theoretical analysis, we establish a connection between our model and diffusion models and show that it includes Bayesian Flow Networks (BFNs) as a special case. In our experiments, we demonstrate that our model improves sample quality on ImageNet32 over both BFNs and the closely related Variational Diffusion Models, while achieving equivalent log-likelihoods on ImageNet32 and CIFAR10. Find our code at https://github.com/martenlienen/bsi.

Figures (8)

• Figure 1: We view generation as the problem of inferring the identity of an unknown sample ${\bm{x}}$ from noisy observations. 1. To begin, our belief about ${\bm{x}}$ is so broad as to cover the complete data distribution. 2. We use a model $f_{{\bm{\theta}}}$ to guess which ${\bm{x}}$ likely corresponds to the information we have collected so far. 3. Now, we pretend that $\mathcolor[rgb]{0.996078431372549,0.7019607843137254,0.03137254901960784}{\hat{{\bm{x}}}}$ is the true ${\bm{x}}$ and take a noisy measurement $\mathcolor[rgb]{0.8274509803921568,0.28627450980392155,0.3058823529411765}{{\bm{y}}}$. 4. We form the posterior belief about ${\bm{x}}$ to incorporate the information contained in $\mathcolor[rgb]{0.8274509803921568,0.28627450980392155,0.3058823529411765}{{\bm{y}}}$. 5. Repeat until we have identified a new sample with sufficient precision $\lambda_{i}$.
• Figure 2: encoders $\mathrm{q}$, i.e. training inputs, of , and . $t$ parametrizes the precision levels by the respective model's precision schedule with $t = 0$ being pure noise, ideally, and $t = 1$ almost equaling the data. Top half shows the mean of $\mathrm{q}$ and bottom half a sample. Mean ${\bm{0}}$ is gray because all models rescale the data to $[-1, 1]$. apply little noise overall and reach a deterministic state at $t = 0$. For , significant information about the sample is preserved in the structure of the mean at the highest noise level. In contrast, converges to its latent prior distribution.
• Figure 3: BSI's sample quality converges quickly and to a lower FID with increasing number of steps.
• Figure 4: $\mathcal{L}_{\mathrm{M}}^{k}$ converges to $\mathcal{L}_{\mathrm{M}}^{\infty}$ from above as predicted in \ref{['thm:elbo-tighter']}.
• Figure 5: Our proposal distribution shrinks the range of $h(\lambda)/\mathrm{p}(\lambda)$, reducing ELBO variance.

Theorems & Definitions (28)

• Lemma 2.1: Posterior Update
• proof
• Theorem 3.1
• Theorem 3.2
• Lemma 3.2
• Lemma 3.2
• Corollary 3.2
• Corollary 3.2
• proof : Proof of \ref{['eq:diffusion-bsi-forward']}
• Lemma D.1: Update Marginal