Debiasing Classifiers by Amplifying Bias with Latent Diffusion and Large Language Models

TL;DR

DiffuBias is introduced, a novel pipeline for text-to-image generation that generates bias-conflict samples, without any training, using pretrained diffusion and image captioning models to debias the classifier.

Abstract

Neural networks struggle with image classification when biases are learned and misleads correlations, affecting their generalization and performance. Previous methods require attribute labels (e.g. background, color) or utilizes Generative Adversarial Networks (GANs) to mitigate biases. We introduce DiffuBias, a novel pipeline for text-to-image generation that enhances classifier robustness by generating bias-conflict samples, without requiring training during the generation phase. Utilizing pretrained diffusion and image captioning models, DiffuBias generates images that challenge the biases of classifiers, using the top-$K$ losses from a biased classifier ($f_B$) to create more representative data samples. This method not only debiases effectively but also boosts classifier generalization capabilities. To the best of our knowledge, DiffuBias is the first approach leveraging a stable diffusion model to generate bias-conflict samples in debiasing tasks. Our comprehensive experimental evaluations demonstrate that DiffuBias achieves state-of-the-art performance on benchmark datasets. We also conduct a comparative analysis of various generative models in terms of carbon emissions and energy consumption to highlight the significance of computational efficiency.

Figures (8)

• Figure 1: Our overall pipeline of DiffuBias. It has four main components: (a), (b), (c), and (d), each executed sequentially. a) First, we intentionally train the classifier on the biased dataset and eventually make a biased classifier and extract bias based on top-$K$ losses. b) We utilize the extracted biased images and make image captions using freezed image encoder and LLM to create text corpus, $\mathcal{C}_T$. c) By leveraging pretrained latent diffusion model, we do text-to-image generation task to amplify bias by generating bias-conflict samples. d) Lastly, we use amplified dataset to train and debias the classifier.
• Figure 2: t-SNE embeddings of BFFHQ using vanilla ResNet-18 and DiffuBias of label young and old. (a) demonstrates that original BFFHQ dataset is heavily biased thus making the vanilla ResNet-18 difficult to classify given the scarce number of bias-conflict samples. (b) and (c) demonstrates t-SNE embeddings of generated and generated + original dataset, respectively.
• Figure 3: Comparative analysis of generative models based on carbon emissions gCO2.eq and computational time (hours) for the BFFHQ dataset. The graphs illustrate gCO2.eq emissions and hours respectively, with disproportionate scales for clarity.
• Figure 4: Generated synthetic bias-conflict samples from our proposed framework, DiffuBias.
• Figure 5: Wrong images generated without text filter, $\mathcal{F_T}$ from BAR captions. The images do not contain any actions. Instead, only background (task-irrelevant feature) part of the text is generated.