Component-Based Fairness in Face Attribute Classification with Bayesian Network-informed Meta Learning

TL;DR

This work introduces face component fairness, a fine-grained fairness notion defined over biological facial features, and proposes Bayesian Network-informed Meta Reweighting (BNMR) to mitigate component-level bias. BNMR combines a fairness-optimized meta-learning reweighting scheme with a Bayesian Network calibrator to model interdependencies among face components and guide per-batch sample weighting, addressing both label scarcity and attribute inter-dependencies. Empirical results on CelebA show BNMR achieves superior fairness metrics (lower $\text{DIG}$ and $\text{TPRD}$) while maintaining competitive accuracy, with stronger gains as more face components are considered; BNMR also exhibits a positive linkage to demographic fairness (e.g., gender). The work highlights face component fairness as a practical surrogate objective for demographic fairness and provides code to facilitate replication and extension.

Abstract

The widespread integration of face recognition technologies into various applications (e.g., access control and personalized advertising) necessitates a critical emphasis on fairness. While previous efforts have focused on demographic fairness, the fairness of individual biological face components remains unexplored. In this paper, we focus on face component fairness, a fairness notion defined by biological face features. To our best knowledge, our work is the first work to mitigate bias of face attribute prediction at the biological feature level. In this work, we identify two key challenges in optimizing face component fairness: attribute label scarcity and attribute inter-dependencies, both of which limit the effectiveness of bias mitigation from previous approaches. To address these issues, we propose \textbf{B}ayesian \textbf{N}etwork-informed \textbf{M}eta \textbf{R}eweighting (BNMR), which incorporates a Bayesian Network calibrator to guide an adaptive meta-learning-based sample reweighting process. During the training process of our approach, the Bayesian Network calibrator dynamically tracks model bias and encodes prior probabilities for face component attributes to overcome the above challenges. To demonstrate the efficacy of our approach, we conduct extensive experiments on a large-scale real-world human face dataset. Our results show that BNMR is able to consistently outperform recent face bias mitigation baselines. Moreover, our results suggest a positive impact of face component fairness on the commonly considered demographic fairness (e.g., \textit{gender}). Our findings pave the way for new research avenues on face component fairness, suggesting that face component fairness could serve as a potential surrogate objective for demographic fairness. The code for our work is publicly available~\footnote{https://github.com/yliuaa/BNMR-FairCompFace.git}.

Figures (6)

• Figure 1: Illustration of face component dependency relations extracted by a Bayesian Network structural searching process in CelebA dataset, where significant dependencies are highlighted with their $\phi$ values from chi-square test of independence. Face component fairness requires a classifier to be fair across various biological face components (e.g., lips, eyebrows, and nose). Achieving fairness at the component level requires unbiased predictions across correlated attributes, as biases in one component can affect others due to their inter-dependencies.
• Figure 2: An overview of our method, which learns a weight vector $w$ with a Bayesian Network calibrator$(\Phi)$ in a fairness-aware meta-learning.
• Figure 3: Weighting sensitivity analysis on $\tau$ in re-weighting. The value $\tau=0.9$ provides the best fairness result. A value of $\tau$ that is either excessively high or low diminishes the effectiveness of bias mitigation efforts.
• Figure 4: Examples of false smiling detections caused by the spurious correlation with "arched eyebrows" across different genders. These cases highlight the presence of face component bias, where the feature "arched eyebrows" disproportionately contributes to incorrect predictions, illustrating the need to address such biases in smiling detection models.
• Figure 5: Bayesian Network structure learned for all face attributes on all images in CelebA dataset. The dense connections represent intricate interdependencies, emphasizing the need for probabilistic reasoning when debiasing with respect to face component attributes.

Theorems & Definitions (2)

• definition 1: True Positive Rate Disparity (TPRD)
• definition 2: Disparate Impact Gap (DIG)