Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models

Raphaël Razafindralambo, Rémy Sun, Frédéric Precioso, Damien Garreau, Pierre-Alexandre Mattei

TL;DR

It is found that while ensembling the scores generally improves the score-matching loss and model likelihood, it fails to consistently enhance perceptual quality metrics such as FID on image datasets.

Abstract

Diffusion models now generate high-quality, diverse samples, with an increasing focus on more powerful models. Although ensembling is a well-known way to improve supervised models, its application to unconditional score-based diffusion models remains largely unexplored. In this work we investigate whether it provides tangible benefits for generative modelling. We find that while ensembling the scores generally improves the score-matching loss and model likelihood, it fails to consistently enhance perceptual quality metrics such as FID on image datasets. We confirm this observation across a breadth of aggregation rules using Deep Ensembles, Monte Carlo Dropout, on CIFAR-10 and FFHQ. We attempt to explain this discrepancy by investigating possible explanations, such as the link between score estimation and image quality. We also look into tabular data through random forests, and find that one aggregation strategy outperforms the others. Finally, we provide theoretical insights into the summing of score models, which shed light not only on ensembling but also on several model composition techniques (e.g. guidance).

When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models

TL;DR

It is found that while ensembling the scores generally improves the score-matching loss and model likelihood, it fails to consistently enhance perceptual quality metrics such as FID on image datasets.

Abstract

Diffusion models now generate high-quality, diverse samples, with an increasing focus on more powerful models. Although ensembling is a well-known way to improve supervised models, its application to unconditional score-based diffusion models remains largely unexplored. In this work we investigate whether it provides tangible benefits for generative modelling. We find that while ensembling the scores generally improves the score-matching loss and model likelihood, it fails to consistently enhance perceptual quality metrics such as FID on image datasets. We confirm this observation across a breadth of aggregation rules using Deep Ensembles, Monte Carlo Dropout, on CIFAR-10 and FFHQ. We attempt to explain this discrepancy by investigating possible explanations, such as the link between score estimation and image quality. We also look into tabular data through random forests, and find that one aggregation strategy outperforms the others. Finally, we provide theoretical insights into the summing of score models, which shed light not only on ensembling but also on several model composition techniques (e.g. guidance).
Paper Structure (76 sections, 8 theorems, 76 equations, 16 figures, 13 tables)

This paper contains 76 sections, 8 theorems, 76 equations, 16 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Preliminary on ensembles and diffusion models
  3. Ensemble methods
  4. Diffusion models
  5. General framework: score-based SDEs
  6. Unifying perspectives on denoising generative processes
  7. How to make an Ensemble of Diffusion models?
  8. Instantiate $K$ score-based predictors
  9. Ensembles of neural networks
  10. Ensembles of decision trees
  11. How to aggregate the $K$ score predictors?
  12. Contextualization of score combinations with other uses
  13. Does averaging scores make sense? A theoretical insight
  14. $K+1$ models are better than $K$ for score estimation
  15. Diffused PoE is not the PoE of diffused distributions
  16. ...and 61 more sections

Key Result

Proposition 4.1

Let ${\bm{s}}_{\bm{\theta}}^{(1)}, \dots, {\bm{s}}_{\bm{\theta}}^{(K)}$ be $K$ score estimators mapping $\mathbb{R}^d \times [0,T]$ to $\mathbb{R}^d$. If for any pair of random variables $({\mathbf{x}}_t, t) \in \mathbb{R}^d \times [0,T]$, the outputs $\{{\bm{s}}^{(1)}_{\bm{\theta}}({\mathbf{x}}_t,t

Figures (16)

  • Figure 1: Visual comparison of samples and FID-10k ($\downarrow$) from two individual models from an ensemble of size $K=4$ trained on FFHQ-256, the ensemble using the arithmetic mean and DDIM (\ref{['section: unifying perspective neural network']}), and the distribution of FID-10k on the ensemble. The initial noise seed is fixed. Ensembling does not clearly improve results: quantitatively, ensembling does not beat the best model. See \ref{['section: averaging fails']} for the evolution of two image quality metrics with respect to $K$.
  • Figure 2: Overview of ensembling within the Diffusion Models framework. We build $K$ score models, for example using Deep Ensemble where each model optimize in parallel the same loss $L_\text{DDSM}$. At inference time, we start from noise and generate ${\mathbf{x}}_0$ using an SDE solver update rule at each step in which we combine the score models using a specific combination rule (e.g. arithmetic mean), instead of using one score model.
  • Figure 3: Evolution of FID ($\downarrow$) and KID ($\downarrow$) and $L_\text{DDSM}$ ($\downarrow$) in function of ensemble size. Samples of the models in \ref{['fig: fid-ffhq', 'fig: kid-ffhq']} are displayed in \ref{['fig: first example']}. $L_\text{DDSM}$ is only evaluated on CIFAR-10 since it corresponds to its training objective. See \ref{['app: experiment details']} for details on each experiment (uncertainty bounds).
  • Figure 4: Predictive diversity across three ensemble methods. Increasing $\lambda$ slightly enhances diversity. DE scale $\lambda$ denotes Deep Ensemble with models trained on initialization scale $\lambda$. See \ref{['app: effect scaling on predictive diversity']} for details on diversity calculation.
  • Figure 5: Comparison of DE ($K{=}2$) and individual model ($K{=}1$) in terms of training loss and FID, plotted against the number of training iterations. The difference stays positive for $L_\text{DDSM}$ (a), while for FID (b), for which we directly plot it, the $\Delta$ varies from positive to negative. See \ref{['app: experiment details']} for details on the CI.
  • ...and 11 more figures

Theorems & Definitions (15)

  • Proposition 4.1: Monotonicity for the DDSM loss
  • Proposition 4.2
  • Proposition C.1
  • proof
  • Definition C.1: Product of Experts (PoE)
  • Lemma C.1
  • proof
  • Proposition C.2
  • proof
  • Lemma C.2: Reverse Young's Inequality for Products
  • ...and 5 more