Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fair Sampling in Diffusion Models through Switching Mechanism

Yujin Choi, Jinseong Park, Hoki Kim, Jaewook Lee, Saerom Park

TL;DR

This work mathematically prove and experimentally demonstrate the effectiveness of the fairness-aware sampling method called attribute switching mechanism for diffusion models on the generation of fair data and the preservation of the utility of the generated data.

Abstract

Diffusion models have shown their effectiveness in generation tasks by well-approximating the underlying probability distribution. However, diffusion models are known to suffer from an amplified inherent bias from the training data in terms of fairness. While the sampling process of diffusion models can be controlled by conditional guidance, previous works have attempted to find empirical guidance to achieve quantitative fairness. To address this limitation, we propose a fairness-aware sampling method called \textit{attribute switching} mechanism for diffusion models. Without additional training, the proposed sampling can obfuscate sensitive attributes in generated data without relying on classifiers. We mathematically prove and experimentally demonstrate the effectiveness of the proposed method on two key aspects: (i) the generation of fair data and (ii) the preservation of the utility of the generated data.

Fair Sampling in Diffusion Models through Switching Mechanism

TL;DR

This work mathematically prove and experimentally demonstrate the effectiveness of the fairness-aware sampling method called attribute switching mechanism for diffusion models on the generation of fair data and the preservation of the utility of the generated data.

Abstract

Diffusion models have shown their effectiveness in generation tasks by well-approximating the underlying probability distribution. However, diffusion models are known to suffer from an amplified inherent bias from the training data in terms of fairness. While the sampling process of diffusion models can be controlled by conditional guidance, previous works have attempted to find empirical guidance to achieve quantitative fairness. To address this limitation, we propose a fairness-aware sampling method called \textit{attribute switching} mechanism for diffusion models. Without additional training, the proposed sampling can obfuscate sensitive attributes in generated data without relying on classifiers. We mathematically prove and experimentally demonstrate the effectiveness of the proposed method on two key aspects: (i) the generation of fair data and (ii) the preservation of the utility of the generated data.
Paper Structure (37 sections, 2 theorems, 19 equations, 17 figures, 5 tables, 2 algorithms)

This paper contains 37 sections, 2 theorems, 19 equations, 17 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary and Related Work
  3. Notations
  4. Fairness Notions
  5. Diffusion Models
  6. Recent Diffusion Sampling
  7. Fair Generative Model
  8. Methodology
  9. Problem Statement
  10. Sampling with Attribute Switching Mechanism
  11. Preserving the distribution of the ODE solution
  12. Fair Sampling with Attribute Switching
  13. Fair Condition of Transition Point $\boldsymbol{\tau}$
  14. Empirical Validation of Transition Points $\boldsymbol{\tau}$
  15. Experiments
  16. ...and 22 more sections

Key Result

Theorem 1

Assuming a pre-trained model is trained with Equation (eq:forward), and the subsequent reverse ODE represents an equivalent probability flow ODE corresponding to the pre-trained model Equation (eq:reverse_ode). Then, the solution of following reverse ODE has the same distribution with the pre-traine where $S_t$ follows Equation (eq:s_t).

Figures (17)

  • Figure 1: Motivation concerning diffusion learning stages and illustration of the proposed method. In transition point $\tau$, the proposed method switches the condition of sensitive attributes from $s_0$ to $s_1$ to generate synthetic data satisfying $\epsilon$-fairness.
  • Figure 2: $\tau$-search algorithm and its efficiency. (a) $\tau$-searching results with diverse diffusion models and datasets and (b) test accuracy of true data evaluated with models trained using synthetic data with varying $\tau$ values.
  • Figure 3: Dimension reduction (PCA, $n=2$) results of the data distribution for both the true data and the generated data of the proposed methods for Bird-Truck, Bird-Frog, and Frog-Truck pairs within the CIFAR-10 datasets.
  • Figure 4: Sampling from stable diffusion with (Top) vanilla sampling and (Bottom) switching sensitive attribute by the proposed method. After switching, the uncontrolled features such as hairstyle, dress, and mustache are maintained.
  • Figure 5: Generated pairwise images with the same random seeds. In each row, images with red borderlines denote vanilla sampling images. Beginning with these vanilla images, we progressively increase the value of $\tau$, moving further away from the initial vanilla image.
  • ...and 12 more figures

Theorems & Definitions (7)

  • Definition 1
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • proof
  • proof