On Fairness of Low-Rank Adaptation of Large Models

TL;DR

This study investigates the fairness implications of LoRA, a parameter-efficient low-rank adaptation method, by comparing it against full-model fine-tuning across vision and language domains. LoRA updates are modeled as $\Delta\mathbf{W} \approx \mathbf{B}\mathbf{A}$ with rank $r$, enabling efficient fine-tuning while preserving base model weights. Across tasks like hatespeech detection, gender/age classification, machine translation, and language modeling on models including ViT-Base, Swin-v2-Large, Llama-2 7B, and Mistral 7B, the results show no consistent pattern of LoRA worsening subgroup fairness; in many cases LoRA is equivalent or even more fair under certain metrics and baselines. The paper highlights the influence of base model quality, task design, and metric choice on fairness outcomes, notes substantial challenges due to generative token biases, and calls for task-aware, robust fairness evaluations and broader comparisons with other parameter-efficient fine-tuning methods.

Abstract

Low-rank adaptation of large models, particularly LoRA, has gained traction due to its computational efficiency. This efficiency, contrasted with the prohibitive costs of full-model fine-tuning, means that practitioners often turn to LoRA and sometimes without a complete understanding of its ramifications. In this study, we focus on fairness and ask whether LoRA has an unexamined impact on utility, calibration, and resistance to membership inference across different subgroups (e.g., genders, races, religions) compared to a full-model fine-tuning baseline. We present extensive experiments across vision and language domains and across classification and generation tasks using ViT-Base, Swin-v2-Large, Llama-2 7B, and Mistral 7B. Intriguingly, experiments suggest that while one can isolate cases where LoRA exacerbates model bias across subgroups, the pattern is inconsistent -- in many cases, LoRA has equivalent or even improved fairness compared to the base model or its full fine-tuning baseline. We also examine the complications of evaluating fine-tuning fairness relating to task design and model token bias, calling for more careful fairness evaluations in future work.

Figures (31)

• Figure 1: LoRA vs. full fine-tuning on group-wise accuracy and equalized odds difference (EOD, lower is fairer) on UTK-Face gender and age classification for ViT-Base (figs 1, 3) and Swin-v2-Large (figs 2, 4). Error bars: 95% CI across 5 seeds. By all metrics LoRA may be considered less fair than full fine-tuning on ViT-Base but equally as fair when switched to a better base model Swin-v2-Large.
• Figure 2: LoRA vs. full fine-tuning on group-wise accuracy, demographic parity difference (DPD, lower is fairer), and equalized odds difference (EOD, lower is fairer). Error bars: 95% CI across five seeds. Numbers in brackets: subgroup sizes. Rows: Llama-2 7B and Mistral 7B on D-Lab religion hatespeech detection. Columns: group-wise accuracy, DPD, EOD. No consistent pattern that LoRA worsens subgroup fairness compared to full fine-tune, and tendency can flip across the base models.
• Figure 3: Confidence histograms (top) and reliability diagrams (bottom) for Llama-2 7B on D-Lab religion (left), Swin-v2-Large on UTK-Face gender (middle), and Llama-2 7B on subgroups with highest ECE on D-Lab religion (right). Dotted purple line indicates perfect calibration. Gap is calculated by confidence minus accuracy. Model with a lower ECE is better calibrated.
• Figure 4: Likelihood Ratio Attack (LiRA) on Swin-v2-Large for membership inference on UTK-Face gender. LoRA models are slightly more resistant to MIA than full fine-tuning.
• Figure 5: Likelihood Ratio Attack (LiRA) on Llama-2 7B for membership inference on D-Lab religion. LoRA models are roughly equally resistant to MIA compared to full fine-tuning.