Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Unbiased Diffusion Models From Biased Dataset

Yeongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, Il-Chul Moon

TL;DR

This work tackles dataset bias in diffusion models by introducing Time-dependent Importance Reweighting (TIW), which estimates a time-varying density ratio between biased and unbiased data under forward diffusion. The TIW-DSM objective combines reweighting with a score-correction term, and is theoretically equivalent to the classical score-matching objective up to a constant, ensuring convergence to the unbiased data distribution. Empirically, TIW-DSM outperforms time-independent baselines across CIFAR-10/100, FFHQ, and CelebA under various bias and reference settings, demonstrating improved sample quality and unbiased latent statistics. The approach enables biased real-world data to be leveraged while steering generated samples toward the true data distribution, with practical impact for fairer generative modeling under weak supervision.

Abstract

With significant advancements in diffusion models, addressing the potential risks of dataset bias becomes increasingly important. Since generated outputs directly suffer from dataset bias, mitigating latent bias becomes a key factor in improving sample quality and proportion. This paper proposes time-dependent importance reweighting to mitigate the bias for the diffusion models. We demonstrate that the time-dependent density ratio becomes more precise than previous approaches, thereby minimizing error propagation in generative learning. While directly applying it to score-matching is intractable, we discover that using the time-dependent density ratio both for reweighting and score correction can lead to a tractable form of the objective function to regenerate the unbiased data density. Furthermore, we theoretically establish a connection with traditional score-matching, and we demonstrate its convergence to an unbiased distribution. The experimental evidence supports the usefulness of the proposed method, which outperforms baselines including time-independent importance reweighting on CIFAR-10, CIFAR-100, FFHQ, and CelebA with various bias settings. Our code is available at https://github.com/alsdudrla10/TIW-DSM.

Training Unbiased Diffusion Models From Biased Dataset

TL;DR

This work tackles dataset bias in diffusion models by introducing Time-dependent Importance Reweighting (TIW), which estimates a time-varying density ratio between biased and unbiased data under forward diffusion. The TIW-DSM objective combines reweighting with a score-correction term, and is theoretically equivalent to the classical score-matching objective up to a constant, ensuring convergence to the unbiased data distribution. Empirically, TIW-DSM outperforms time-independent baselines across CIFAR-10/100, FFHQ, and CelebA under various bias and reference settings, demonstrating improved sample quality and unbiased latent statistics. The approach enables biased real-world data to be leveraged while steering generated samples toward the true data distribution, with practical impact for fairer generative modeling under weak supervision.

Abstract

With significant advancements in diffusion models, addressing the potential risks of dataset bias becomes increasingly important. Since generated outputs directly suffer from dataset bias, mitigating latent bias becomes a key factor in improving sample quality and proportion. This paper proposes time-dependent importance reweighting to mitigate the bias for the diffusion models. We demonstrate that the time-dependent density ratio becomes more precise than previous approaches, thereby minimizing error propagation in generative learning. While directly applying it to score-matching is intractable, we discover that using the time-dependent density ratio both for reweighting and score correction can lead to a tractable form of the objective function to regenerate the unbiased data density. Furthermore, we theoretically establish a connection with traditional score-matching, and we demonstrate its convergence to an unbiased distribution. The experimental evidence supports the usefulness of the proposed method, which outperforms baselines including time-independent importance reweighting on CIFAR-10, CIFAR-100, FFHQ, and CelebA with various bias settings. Our code is available at https://github.com/alsdudrla10/TIW-DSM.
Paper Structure (46 sections, 4 theorems, 38 equations, 31 figures, 10 tables, 2 algorithms)

This paper contains 46 sections, 4 theorems, 38 equations, 31 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Setup
  4. Diffusion model and score matching
  5. Density ratio estimation
  6. Importance reweighting for unbiased generative learning
  7. Method
  8. Why time-dependent importance reweighting?
  9. Score matching with time-dependent importance reweighting
  10. Experiments
  11. Latent Bias on the class
  12. Latent bias on sensitive attributes
  13. Ablation Studies
  14. Loss component
  15. Density ratio scaling
  16. ...and 31 more sections

Key Result

Theorem 1

$\mathcal{L}_{\text{TIW-DSM}}(\boldsymbol{\theta};p_{\text{bias}}, w_{\boldsymbol{\phi}^*}^t(\cdot))$ = $\mathcal{L}_{\text{SM}}(\boldsymbol{\theta};p_{\text{data}})+C$, where $C$ is a constant w.r.t. $\boldsymbol{\theta}$.

Figures (31)

  • Figure 1: The samples that reflect the proportion of four latent subgroups. The proposed method mitigates the latent bias statistics as shown in (b).
  • Figure 2: Accuracy of density ratio estimation between $p_{\text{bias}}$ and $p_{\text{data}}$ under diffusion process. (a-b) Samples from two distributions. (c-d) Density ratio statistics on the ground truth and the model, at each diffusion time. (e) Density ratio estimation error according to $t$. The density ratio error becomes significantly decreases as $t$ becomes larger.
  • Figure 3: (a-b) The score plots on $p^0_{\text{bias}}$ and $p^0_{\text{data}}$ defined in \ref{['fig:2']}. (c) The score plot on score correction term. (d) The reweighting value. The time-dependent density ratio simultaneously mitigates the bias through (c) and (d).
  • Figure 4: Analysis on CIFAR-10 (LT / 5%) experiments. (a-d) Samples that reflect the diversity and latent statistics with (FID / Recall). (e) Training curves for each method.
  • Figure 5: The convergence of TIW-DSM on various bias level & reference size.
  • ...and 26 more figures

Theorems & Definitions (6)

  • Theorem 1
  • Corollary 2
  • Theorem 2
  • proof
  • Theorem 3
  • proof