Harnessing Diffusion-Generated Synthetic Images for Fair Image Classification

TL;DR

This work tackles biases in image classification caused by imbalanced training data by leveraging diffusion-generated, group-balanced images. It compares Vanilla Stable Diffusion, LoRA-based fine-tuning, DreamBooth, and Clustered DreamBooth within a two-stage pipeline that pretrains on synthetic data and then finetunes on real biased data, complemented by CLIP-based filtering and a combined CE+SupCon loss. The main contribution is the introduction of Clustered DreamBooth to capture intra-group diversity and its empirical demonstration that diffusion-based pretraining yields competitive or superior fairness performance, especially under severe bias, relative to state-of-the-art debiasing methods. The work also analyzes design choices such as cluster counts, CLIP weighting, and loss composition, providing qualitative evidence of distribution-aligned synthetic images. Limitations include training complexity and some domain-specific failure modes, pointing to future work on optimization and integration with other debiasing techniques.

Abstract

Image classification systems often inherit biases from uneven group representation in training data. For example, in face datasets for hair color classification, blond hair may be disproportionately associated with females, reinforcing stereotypes. A recent approach leverages the Stable Diffusion model to generate balanced training data, but these models often struggle to preserve the original data distribution. In this work, we explore multiple diffusion-finetuning techniques, e.g., LoRA and DreamBooth, to generate images that more accurately represent each training group by learning directly from their samples. Additionally, in order to prevent a single DreamBooth model from being overwhelmed by excessive intra-group variations, we explore a technique of clustering images within each group and train a DreamBooth model per cluster. These models are then used to generate group-balanced data for pretraining, followed by fine-tuning on real data. Experiments on multiple benchmarks demonstrate that the studied finetuning approaches outperform vanilla Stable Diffusion on average and achieve results comparable to SOTA debiasing techniques like Group-DRO, while surpassing them as the dataset bias severity increases.

Figures (13)

• Figure 1: Generation Approaches. In this paper, we investigate four image generation pipelines for training fairer classifiers: a) Vanilla SD, which only accepts prompts, b) LoRA-finetuning, which finetunes the diffusion model on the images of a group, c) Dreambooth, which finetunes the diffusion model and represents the images of a group using a special token 'V', d) Clustered Dreambooth, which finetunes a diffusion model on different clusters $k_i$ present in the training group, representing each of them through a special token 'V'.
• Figure 2: Overview of the studied pipeline. In Stage 1, we generate images uniformly from each group (e.g., non-blond female ($g_1$), non-blond male ($g_2$), blond female ($g_3$), blond male ($g_4$)) using the generatice approaches, and train a classification model $f^{\text{pre}}$ with CE and SupCon losses. In Stage 2, we finetune only the linear classifier on the original dataset.
• Figure 3: Image Generation Pipeline for Bias Ratio$\mathbf{=0.999}$. Bias-conflicting samples (e.g., Blond Males in CelebA) are generated using models trained on bias-aligned images (e.g., Blond Females).
• Figure 4: Training vs Generated Images from Clustered Dreambooth. In this figure, we present original images from each group in the studied datasets (images with red border) along with the generated ones (yellow border). We note how the generated images closely reflect the training distribution and the group-related attributes.
• Figure 5: Dreambooth vs Clustered Dreambooth for UTKFace Female Children. Clustered Dreambooth samples are more diverse than the Dreambooth ones.