DiffInject: Revisiting Debias via Synthetic Data Generation using Diffusion-based Style Injection

TL;DR

DiffInject tackles dataset bias in vision tasks by proposing an unsupervised diffusion-based data augmentation strategy. It identifies bias-conflict samples as top-$K$ losses from an overfit biased classifier, then employs a diffusion model with Perception Prioritized weighting to synthesize bias-conflict content, which is injected into images via latent-space Slerp-based content manipulation during the DDIM reverse process. Training an unbiased classifier on the combined synthetic and original data yields improvements over strong baselines and achieves state-of-the-art on Colored MNIST, while revealing limitations on CCIFAR-10. Overall, DiffInject demonstrates that diffusion models can be leveraged for unsupervised debiasing without explicit bias labels, offering a scalable path for mitigating shortcuts in visual datasets.

Abstract

Dataset bias is a significant challenge in machine learning, where specific attributes, such as texture or color of the images are unintentionally learned resulting in detrimental performance. To address this, previous efforts have focused on debiasing models either by developing novel debiasing algorithms or by generating synthetic data to mitigate the prevalent dataset biases. However, generative approaches to date have largely relied on using bias-specific samples from the dataset, which are typically too scarce. In this work, we propose, DiffInject, a straightforward yet powerful method to augment synthetic bias-conflict samples using a pretrained diffusion model. This approach significantly advances the use of diffusion models for debiasing purposes by manipulating the latent space. Our framework does not require any explicit knowledge of the bias types or labelling, making it a fully unsupervised setting for debiasing. Our methodology demonstrates substantial result in effectively reducing dataset bias.

Figures (6)

• Figure 1: Overall framework of our proposed method, DiffInject.
• Figure 2: Generated bias-conflict samples with DiffInject. The three columns represent samples from the original dataset, top-$k$ loss samples and generated samples, respectively.
• Figure 3: Generated bias-conflict samples with artifacts from our framework, DiffInject.
• Figure 4: Examples of generated bias-conflict samples with DiffInject for CMNIST and CCIFAR-10 dataset. The three columns represent samples from the original dataset, top-$k$ loss samples and generated samples, respectively.
• Figure 5: Examples of generated bias-conflict samples with DiffInject for BFFHQ dataset. The three columns represent samples from the original dataset, top-$k$ loss samples and generated samples, respectively.