AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data

TL;DR

AIM-Fair tackles fairness in facial attribute classification when demographic labels are unavailable by integrating Contextual Synthetic Data Generation with a latent diffusion model, and a gradient-based Selective Fine-Tuning framework that updates only bias- and domain-insensitive parameters. The three-component pipeline (CSDG, SMG, SFT) uses GPT-4-driven prompts to produce diverse, label-free synthetic data and identifies a parameter subset via gradient differentials across biased real, biased synthetic, and unbiased synthetic data, enabling fairer yet utility-preserving fine-tuning. Empirical results on CelebA and UTKFace show AIM-Fair achieving superior fairness (lower EO, higher WST) while maintaining or improving ACC compared with fully and partially fine-tuned baselines and state-of-the-art generative-data approaches like DiGA. The work demonstrates a scalable, annotation-free path to debiasing FAC models through context-rich synthetic data and principled, parameter-wise updates, with broad implications for fair AI systems using synthetic data.

Abstract

Recent advances in generative models have sparked research on improving model fairness with AI-generated data. However, existing methods often face limitations in the diversity and quality of synthetic data, leading to compromised fairness and overall model accuracy. Moreover, many approaches rely on the availability of demographic group labels, which are often costly to annotate. This paper proposes AIM-Fair, aiming to overcome these limitations and harness the potential of cutting-edge generative models in promoting algorithmic fairness. We investigate a fine-tuning paradigm starting from a biased model initially trained on real-world data without demographic annotations. This model is then fine-tuned using unbiased synthetic data generated by a state-of-the-art diffusion model to improve its fairness. Two key challenges are identified in this fine-tuning paradigm, 1) the low quality of synthetic data, which can still happen even with advanced generative models, and 2) the domain and bias gap between real and synthetic data. To address the limitation of synthetic data quality, we propose Contextual Synthetic Data Generation (CSDG) to generate data using a text-to-image diffusion model (T2I) with prompts generated by a context-aware LLM, ensuring both data diversity and control of bias in synthetic data. To resolve domain and bias shifts, we introduce a novel selective fine-tuning scheme in which only model parameters more sensitive to bias and less sensitive to domain shift are updated. Experiments on CelebA and UTKFace datasets show that our AIM-Fair improves model fairness while maintaining utility, outperforming both fully and partially fine-tuned approaches to model fairness.

Figures (8)

• Figure 1: Facial attributes classification (FAC) on the CelebA dataset based on different training strategies, in which Smiling is the target attribute and Gender is the protected attribute. This shows different learning strategies result in variable model utility (Overall Accuracy) vs. model fairness (Worst Group Accuracy and Equalized Odds) on demographic groups. A model trained solely on real data if biased exhibits high accuracy but poor fairness scores. Conversely, models trained on balanced synthetic data show better fairness but poorer accuracy due to a "domain gap" between the real and synthetic data, and a lack of selective model fine-tuning when synthetic data is deployed. Strategies to repair imbalances in real data ye2023exploitingd2024improving or to supplement real data with synthetic data shin2023fill marginally increase accuracy but do little to improve fairness. Our method for selective fine-tuning of a pre-trained (biased) model with synthetic data not only preserves model accuracy but also substantially improves fairness, outperforming fully fine-tuning (FFT) in both model utility and fairness.
• Figure 2: The effects of selective fine-tuning by layer-wise freezing from facial attribute classifications on the CelebA dataset, with Smiling as the target attribute and Male as the protected attribute: When only the fully connected (FC) layer is frozen the model shows improved worst demographic group accuracy and reduced equalized odds, i.e. more fair, but sacrifices some utility (overall accuracy). This indicates increased cross-domain generalisability (real and synthetic). Conversely, freezing only block 2 while fine-tuning the remaining parameters results in high overall accuracy but poorer fairness, i.e. further enhanced domain-bias specificity. When only block 1 is frozen, the model not only maintained equalized odds but also increased utility (overall accuracy) and worst group accuracies.
• Figure 3: A selective fine-tuning model consisting of three parts: (1) Contextual Synthetic Data Generation (CSDG) for generating diverse images using GPT-4 generated prompts, (2) Selective Mask Generation (SMG) for creating a selection mask that determines which parameters are updated during fine-tuning, and (3) Selective Fine-Tuning (SFT) to enhance model fairness obtained from synthetic data whilst simultaneously to preserve model utility yielded from real data in pre-training.
• Figure 4: Generated images of Smiling Male conditioned on different prompts. Compared to the plain prompt, our contextual prompts enhance the diversity. More on UTKFace in Appendix.
• Figure 5: Classification results on the CelebA dataset (T=Smiling, P=Male) with different top-k values. The top-k values {40, 45, 50, 55, 60} correspond approximately to {64%, 72%, 80%, 88%, 96%} of the total model parameters.