Dissecting and Mitigating Diffusion Bias via Mechanistic Interpretability

TL;DR

This work tackles social bias in diffusion models by proposing DiffLens, a mechanistic interpretability framework that discovers bias-generating internal mechanisms and enables targeted debiasing. It unfolds through three stages: disentangling diffusion activations into a sparse, monosemantic space with a k-Sparse Autoencoder, identifying bias features with a gradient-based attribution, and intervening in those features via controlled scaling or addition to steer generation while preserving non-target semantics. Across unconditional and text-to-image diffusion models, DiffLens achieves strong bias mitigation (lower Fairness Discrepancy) and maintains image quality and semantic coherence (FID, CLIP-I, CLIP-T), outperforming several baselines. The method provides interpretable insights into how internal diffusion features encode bias and offers fine-grained, controllable edits, with potential to extend to other bias types and architectures. These findings contribute to safer and more controllable generative AI deployment by linking bias mitigation to intrinsic model mechanisms.

Abstract

Diffusion models have demonstrated impressive capabilities in synthesizing diverse content. However, despite their high-quality outputs, these models often perpetuate social biases, including those related to gender and race. These biases can potentially contribute to harmful real-world consequences, reinforcing stereotypes and exacerbating inequalities in various social contexts. While existing research on diffusion bias mitigation has predominantly focused on guiding content generation, it often neglects the intrinsic mechanisms within diffusion models that causally drive biased outputs. In this paper, we investigate the internal processes of diffusion models, identifying specific decision-making mechanisms, termed bias features, embedded within the model architecture. By directly manipulating these features, our method precisely isolates and adjusts the elements responsible for bias generation, permitting granular control over the bias levels in the generated content. Through experiments on both unconditional and conditional diffusion models across various social bias attributes, we demonstrate our method's efficacy in managing generation distribution while preserving image quality. We also dissect the discovered model mechanism, revealing different intrinsic features controlling fine-grained aspects of generation, boosting further research on mechanistic interpretability of diffusion models.

Figures (19)

• Figure 1: Framework of DiffLens exploring the inner working of diffusion models for bias mitigation. Neurons that are activated in the semantic space are defined as fired features and inactivated ones as unfired features. Target features are either suppressed or amplified to control bias level. We divide the framework into three parts (a), (b) and (c) which correspond to \ref{['Sec:disentangle', 'Sec:identify', 'Sec:intervene']}, respectively.
• Figure 2: Comparison of randomly sampled original and debiased images from various baseline methods. The minority group (male) is highlighted with red bounding boxes for easier viewing. "M:F" refers to the male-to-female ratio. Our DiffLens effectively mitigates bias while preserving high generation quality, whereas other methods either struggle to maintain balance or produce images with artifacts.
• Figure 3: Comparison in accurate identification of bias features. Our DiffLens preserves overall image semantics such as smile and eyeglasses while other methods frequently introduce distortions or lose important details.
• Figure 4: Achievable range of bias level control for each method. The Log Gender Ratio reflects the log of male to female ratio in the generated images, with 0 indicating balance. Our DiffLens offers broader bias control, delivering higher generation quality (left plot) and improved semantic coherence (right plot).
• Figure 5: Comparison of individual image transformations along the gender axis, with columns displaying images sampled from progressively shifting ratios as in \ref{['figure:contcurve']}. Our DiffLens achieves smooth and consistent transitions that preserve semantic features like facial expressions, while other methods show distortions and loss of detail at higher imbalance ratios.