Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FairMT: Fairness for Heterogeneous Multi-Task Learning

Guanyu Hu, Tangzheng Lian, Na Yan, Dimitrios Kollias, Xinyu Yang, Oya Celiktutan, Siyang Song, Zeyu Fu

TL;DR

FairMT tackles fairness in multi-task learning with heterogeneous outputs and incomplete supervision by introducing an Asymmetric Heterogeneous Fairness Disparity Aggregation (AHFDA) mechanism and a head-aware optimization proxy. The framework unifies diverse fairness notions across detection, classification, and regression through a directional, worst-case-focused aggregation and a Pareto-guided primal–dual formulation. Empirical results across vision and language benchmarks show substantial fairness gains without sacrificing utility, often surpassing state-of-the-art baselines. The approach also provides scalable optimization via projection-free updates and a head-aware QP proxy, making it practical for real-world, partially labeled MTL problems.

Abstract

Fairness in machine learning has been extensively studied in single-task settings, while fair multi-task learning (MTL), especially with heterogeneous tasks (classification, detection, regression) and partially missing labels, remains largely unexplored. Existing fairness methods are predominantly classification-oriented and fail to extend to continuous outputs, making a unified fairness objective difficult to formulate. Further, existing MTL optimization is structurally misaligned with fairness: constraining only the shared representation, allowing task heads to absorb bias and leading to uncontrolled task-specific disparities. Finally, most work treats fairness as a zero-sum trade-off with utility, enforcing symmetric constraints that achieve parity by degrading well-served groups. We introduce FairMT, a unified fairness-aware MTL framework that accommodates all three task types under incomplete supervision. At its core is an Asymmetric Heterogeneous Fairness Constraint Aggregation mechanism, which consolidates task-dependent asymmetric violations into a unified fairness constraint. Utility and fairness are jointly optimized via a primal--dual formulation, while a head-aware multi-objective optimization proxy provides a tractable descent geometry that explicitly accounts for head-induced anisotropy. Across three homogeneous and heterogeneous MTL benchmarks encompassing diverse modalities and supervision regimes, FairMT consistently achieves substantial fairness gains while maintaining superior task utility. Code will be released upon paper acceptance.

FairMT: Fairness for Heterogeneous Multi-Task Learning

TL;DR

FairMT tackles fairness in multi-task learning with heterogeneous outputs and incomplete supervision by introducing an Asymmetric Heterogeneous Fairness Disparity Aggregation (AHFDA) mechanism and a head-aware optimization proxy. The framework unifies diverse fairness notions across detection, classification, and regression through a directional, worst-case-focused aggregation and a Pareto-guided primal–dual formulation. Empirical results across vision and language benchmarks show substantial fairness gains without sacrificing utility, often surpassing state-of-the-art baselines. The approach also provides scalable optimization via projection-free updates and a head-aware QP proxy, making it practical for real-world, partially labeled MTL problems.

Abstract

Fairness in machine learning has been extensively studied in single-task settings, while fair multi-task learning (MTL), especially with heterogeneous tasks (classification, detection, regression) and partially missing labels, remains largely unexplored. Existing fairness methods are predominantly classification-oriented and fail to extend to continuous outputs, making a unified fairness objective difficult to formulate. Further, existing MTL optimization is structurally misaligned with fairness: constraining only the shared representation, allowing task heads to absorb bias and leading to uncontrolled task-specific disparities. Finally, most work treats fairness as a zero-sum trade-off with utility, enforcing symmetric constraints that achieve parity by degrading well-served groups. We introduce FairMT, a unified fairness-aware MTL framework that accommodates all three task types under incomplete supervision. At its core is an Asymmetric Heterogeneous Fairness Constraint Aggregation mechanism, which consolidates task-dependent asymmetric violations into a unified fairness constraint. Utility and fairness are jointly optimized via a primal--dual formulation, while a head-aware multi-objective optimization proxy provides a tractable descent geometry that explicitly accounts for head-induced anisotropy. Across three homogeneous and heterogeneous MTL benchmarks encompassing diverse modalities and supervision regimes, FairMT consistently achieves substantial fairness gains while maintaining superior task utility. Code will be released upon paper acceptance.

Paper Structure

This paper contains 14 sections, 29 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. FairMT Framework
  4. Asymmetric Heterogeneous Fairness Disparity Aggregation (AHFDA)
  5. Fair Binary Detection.
  6. Fair Multi-class Classification.
  7. Fair Regression.
  8. Fairness Disparity Aggregation
  9. FairMT Framework
  10. FairMT Optimization
  11. Head-Aware Optimization Proxy.
  12. Experiments
  13. Ablation Study and Further Analysis
  14. Conclusion

Figures (3)

  • Figure 1: Comparison of FairMT with Conventional Fair or MTL Paradigms. (a) Our proposed FairMT framework. (b) Comparison between FairMT and conventional paradigms: the left panel highlights our advantages, and the right panel illustrates how AHFDA achieves fairness by asymmetrically lifting only violated groups without harming advantaged groups. (c) Conventional fairness learning paradigms, all applying symmetric penalties that simultaneously suppress advantaged groups and lift disadvantaged groups. (d) Conventional multi-task learning paradigms, both lacking fairness awareness and unable to model head-induced geometry.
  • Figure 2: Comparison between FairMT and state-of-the-art baselines across three multi-task settings: (a) Attribute Detection, (b) Affective Analysis, and (c) Toxicity & Social Bias Analysis.
  • Figure 3: Ablation on heterogeneous AU2--Arousal tasks. We compare conventional MTL, utility-weighted baselines, MGDA, and their AHFDA-enhanced variants on both utility (left) and fairness (right).