FairBranch: Mitigating Bias Transfer in Fair Multi-task Learning

TL;DR

FairBranch tackles negative transfer and bias transfer in fair multi-task learning by introducing parameter-similarity based branching to group related tasks into branches and applying a targeted fairness gradient correction within each branch. This approach reduces accuracy gradient conflicts while mitigating cross-task fairness conflicts, enabling scalable handling of many tasks. Empirical results on ACS-PUMS and CelebA show FairBranch outperforming state-of-the-art MTL methods on both accuracy and fairness, with consistent positive average knowledge gain $\bar{KG}>0$ and non-positive average discrimination gain $\bar{DG}\le 0$. The work provides a principled, architecture-based solution with theoretical intuition for parameter-space grouping and practical benefits for fairness-aware multi-task learning.

Abstract

The generalisation capacity of Multi-Task Learning (MTL) suffers when unrelated tasks negatively impact each other by updating shared parameters with conflicting gradients. This is known as negative transfer and leads to a drop in MTL accuracy compared to single-task learning (STL). Lately, there has been a growing focus on the fairness of MTL models, requiring the optimization of both accuracy and fairness for individual tasks. Analogously to negative transfer for accuracy, task-specific fairness considerations might adversely affect the fairness of other tasks when there is a conflict of fairness loss gradients between the jointly learned tasks - we refer to this as Bias Transfer. To address both negative- and bias-transfer in MTL, we propose a novel method called FairBranch, which branches the MTL model by assessing the similarity of learned parameters, thereby grouping related tasks to alleviate negative transfer. Moreover, it incorporates fairness loss gradient conflict correction between adjoining task-group branches to address bias transfer within these task groups. Our experiments on tabular and visual MTL problems show that FairBranch outperforms state-of-the-art MTLs on both fairness and accuracy.

Figures (7)

• Figure 1: Fairness loss gradient conflicts observed in state-of-the-art MTLs addressing negative transfer of accuracy: (a) TAG Google2021MTLG using task-grouping and (b) Recon guangyuan2022recon using gradient correction on the ACS-PUMS Census Data 2018.
• Figure 2: A High Level Depiction of Branch Formation
• Figure 3: Example Showing Effect of Fairness Gradient Correction on Task-grouped Branches.
• Figure 4: Comparison on Knowledge Gain (KG) and Discrimination Gain (DG) Distribution: Each box provides comparison on a given Metric Labelled on Top. In boxes every triangle depicts Difference between an MTL with Task Specific STLs. Red Triangles indicates Negative/Bias Transfer and Green indicates Positive/Unbiased Gain. Positive Difference for Accuracy, Negative for Fairness are better.
• Figure 5: Accuracy and Fairness Loss Gradient Conflicts of FairBranch over Training Epochs. Each Box shows Distribution of Angle of Conflict Observed at an Epoch. Less Densely Crowded Lower Boxes are Better.