MoESD: Mixture of Experts Stable Diffusion to Mitigate Gender Bias

TL;DR

This paper proposes MoESD (Mixture of Experts Stable Diffusion) with BiAs (Bias Adapters) to mitigate gender bias in text-to-image models and demonstrates that introducing an arbitrary special token to the prompt is essential during the mitigation process.

Abstract

Text-to-image models are known to propagate social biases. For example, when prompted to generate images of people in certain professions, these models tend to systematically generate specific genders or ethnicities. In this paper, we show that this bias is already present in the text encoder of the model and introduce a Mixture-of-Experts approach by identifying text-encoded bias in the latent space and then creating a Bias-Identification Gate mechanism. More specifically, we propose MoESD (Mixture of Experts Stable Diffusion) with BiAs (Bias Adapters) to mitigate gender bias in text-to-image models. We also demonstrate that introducing an arbitrary special token to the prompt is essential during the mitigation process. With experiments focusing on gender bias, we show that our approach successfully mitigates gender bias while maintaining image quality.

Figures (9)

• Figure 1: The Architecture of BiAs. We apply BiAs to the cross-attention of the U-Net. In the figure, the left BiAs represents the female expert, the right one, the male expert, and the middle one, the original cross-attention. The conditional information is processed by the Bias Identification Gate to determine which experts to choose - either male, female, or none of them. In the end, the chosen BiAs will process the input and be added together with the original cross-attention. The detailed BiAs architecture includes BiAs of the Q, K, V, and output matrices (for simplicity, we omit them in the figure). The parameters in cross-attention are frozen during the parameter-efficient fine-tuning process.
• Figure 2: Architecture of our MoE system. We modify rombach2022high by adding our Bias Identification Gate and Bias Adapter Experts. See Section \ref{['sec:MoE']} for details.
• Figure 3: Successful Mitigation of Gender Bias. From left to right are the occupations of aerospace engineer, metal worker, plumber, executive assistant, nurse, and fitness instructor. Each column is generated by the same prompt and seed. The left three are extremely male-biased occupations, and the right three are extremely female-biased occupations. The BiAs Expert successfully leads to a fairer generation.
• Figure 4: T-Distributed Stochastic Neighbor Embedding (T-SNE) dimension reduction and visualization of prompts encoded by text encoder of Stable Diffusion Version 1.5 and 2.1 rombach2022high. There is a clear boundary between two gender-bias occupation embeddings in both versions of Stable Diffusion.
• Figure 5: The number of correct predictions for Stable Diffusion 1.5.