Unbiased Image Synthesis via Manifold Guidance in Diffusion Models

TL;DR

The paper tackles bias in diffusion-based image synthesis by proposing Manifold Guidance Sampling (MGS), an unsupervised, plug-and-play approach that steers sampling toward the true data manifold without retraining. It introduces a two-part framework: (1) data manifold evaluation to learn a manifold-preserving embedding via F and g, and (2) a manifold-guided reverse diffusion step that penalizes deviations from the estimated manifold using a gradient term. Theoretical results connect the embedding objective to rate-reduction-like properties, while experiments across six datasets demonstrate reduced bias (closer attribute distributions to the training data) and improved FID/sFID, with robust ablations validating the method. This approach offers a practical, unsupervised path to enhance diversity and unbiasedness in diffusion-based image synthesis with broad applicability to pre-trained models.

Abstract

Diffusion Models are a potent class of generative models capable of producing high-quality images. However, they often inadvertently favor certain data attributes, undermining the diversity of generated images. This issue is starkly apparent in skewed datasets like CelebA, where the initial dataset disproportionately favors females over males by 57.9%, this bias amplified in generated data where female representation outstrips males by 148%. In response, we propose a plug-and-play method named Manifold Guidance Sampling, which is also the first unsupervised method to mitigate bias issue in DDPMs. Leveraging the inherent structure of the data manifold, this method steers the sampling process towards a more uniform distribution, effectively dispersing the clustering of biased data. Without the need for modifying the existing model or additional training, it significantly mitigates data bias and enhances the quality and unbiasedness of the generated images.

Figures (16)

• Figure 1: (a) Images generated by DDPM using standard sampling (left), and manifold-guided sampling (Ours) (right), from the same model. (b) Gender distribution in the CelebA dataset (left), generated data from DDPM using standard sampling (middle), and manifold-guided sampling (Ours) (right).
• Figure 2: The sampling process of DDPMs at step $t$. ① Unconditional reverse diffusion generates $\boldsymbol{x}_t^i, i=1,2,...,n$. ② Generate $\hat{\boldsymbol{x}}_0^i$ from $\boldsymbol{x}_t^i$ based on Tweedie's formula. ③ Calculate the manifold constraint gradient, where $\boldsymbol{F}$ and $\boldsymbol{g}$ are pre-trained networks. ④ Apply the manifold constraint guidance on the sampling process of DDPMs.
• Figure 3: Distribution of the number of MNIST samples with $k$ generated neighbors within an $\varepsilon$-ball radius. $N$ samples are generated using DDPM model (left) and our method (right). $\varepsilon$ is the average nearest neighbor distance for the MNIST samples.
• Figure 4: "Smile" (a) and "Beard" (b) distribution in the CelebA dataset (left), generated data from DDPM using standard sampling (middle), and manifold-guided sampling (right).
• Figure 5: Distribution of the number of MNIST training samples with $k$ neighbors generated within a 0.8$\varepsilon$-ball radius. Left shows the results of DDPM. Right shows the results of our method.

Theorems & Definitions (6)

• Proposition 1
• Theorem 1
• Proposition 2
• Proof 1
• Theorem 2
• Proof 2