FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation

TL;DR

FairLoRA is introduced, a novel fairness-specific regularizer for LoRA aimed at reducing performance disparities across data subgroups by minimizing per-class variance in loss and is the first to introduce a fairness based finetuning through LoRA.

Abstract

Recent advances in parameter-efficient fine-tuning methods, such as Low Rank Adaptation (LoRA), have gained significant attention for their ability to efficiently adapt large foundational models to various downstream tasks. These methods are appreciated for achieving performance comparable to full fine-tuning on aggregate-level metrics, while significantly reducing computational costs. To systematically address fairness in LLMs previous studies fine-tune on fairness specific data using a larger LoRA rank than typically used. In this paper, we introduce FairLoRA, a novel fairness-specific regularizer for LoRA aimed at reducing performance disparities across data subgroups by minimizing per-class variance in loss. To the best of our knowledge, we are the first to introduce a fairness based finetuning through LoRA. Our results demonstrate that the need for higher ranks to mitigate bias is not universal; it depends on factors such as the pre-trained model, dataset, and task. More importantly, we systematically evaluate FairLoRA across various vision models, including ViT, DiNO, and CLIP, in scenarios involving distribution shifts. We further emphasize the necessity of using multiple fairness metrics to obtain a holistic assessment of fairness, rather than relying solely on the metric optimized during training.

Figures (12)

• Figure 1: We compare the CLIP models with and without fairness regularizers for both full finetuning (FFT) as well as LoRA. Dataset: GeoDE. On the left. we notice that FairLoRA has better overall performance compared to LoRA. On the right, we visualize the effect on the variance of loss across classes and FairLoRA has a lower variance compared to all other methods. More detailed results and comparisons can be found in \ref{['tab:eval']}.
• Figure 2: Comparison of model performance and fairness on the GeoDE across different LoRA ranks as well as FFT. Please note that this doesn't include any specific fairness related intervention. Rank = -1 implies Full-Finetuning.
• Figure 3: Comparison of FairLoRA performance in Clip model across Aircrafts and Waterbirds datasets.
• Figure 4: All metrics are normalized to the same scale and adjusted such that higher is better. Comparison of FairLoRA performance on GeoDE across metrics on CLIP and DiNO.
• Figure 5: Comparison on the impact of rank on performance as well as fairness across models for GeoDe. Higher value is better in both the graphs. There is no monotonic behaviour on fairness or performance when changing the ranks. FairLoRA versions are more stable to changes in ranks.