On the "Illusion" of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes

TL;DR

The paper tackles gender bias in face recognition by moving beyond demographic factors to decorrelated non-demographic attributes. It introduces iGARBE as a Gini-based fairness metric and CoFair for contextual fairness, and develops a greedy, unsupervised framework to identify attribute combinations that eliminate the gender gap when test data share certain attributes. Through decorrelation of 40 attributes using MAAD-Face and evaluations on ArcFace and FaceNet, the authors show that balancing test sets on hair, facial hair, and occluding accessories can largely remove gender-based performance differences with minimal or acceptable impact on overall accuracy. The work highlights the social-behavioral origins of bias and provides a scalable methodology and metrics for analyzing and mitigating fairness issues in face biometrics, with practical implications for dataset design and system robustness.

Abstract

Face recognition systems (FRS) exhibit significant accuracy differences based on the user's gender. Since such a gender gap reduces the trustworthiness of FRS, more recent efforts have tried to find the causes. However, these studies make use of manually selected, correlated, and small-sized sets of facial features to support their claims. In this work, we analyse gender bias in face recognition by successfully extending the search domain to decorrelated combinations of 40 non-demographic facial characteristics. First, we propose a toolchain to effectively decorrelate and aggregate facial attributes to enable a less-biased gender analysis on large-scale data. Second, we introduce two new fairness metrics to measure fairness with and without context. Based on these grounds, we thirdly present a novel unsupervised algorithm able to reliably identify attribute combinations that lead to vanishing bias when used as filter predicates for balanced testing datasets. The experiments show that the gender gap vanishes when images of male and female subjects share specific attributes, clearly indicating that the issue is not a question of biology but of the social definition of appearance. These findings could reshape our understanding of fairness in face biometrics and provide insights into FRS, helping to address gender bias issues.

Figures (4)

• Figure 1: Attribute annotation correlations - The correlations are computed using the Pearson coefficient. The depicted attributes are selected such that the 15.0 highest absolute, i.e., positive or negative, correlations are visible. As can be seen, very strong correlations exist between various attributes, indicating that they are not statistically independent.
• Figure 2: Evolution of critical clustering metrics over progressing decorrelation - The chosen measurements reflect the efficacy of the devised algorithm in reducing the correlation of MAAD-Face's non-demographic attributes. The iteration range reflects the clustering intensity, ranging from 0.0 (no clustering) to 39.0 (all attributes in one cluster). The number of clusters, mean and maximum correlation decrease linearly. For an optimal clustering w.r.t. low inter-cluster correlation and low number of clusters, it is therefore sensible to choose as late of an iteration as possible. To also accommodate the sampling requirements into these criteria, we conceive iteration 13.0 as optimal, since these requirements cannot be fulfilled thereafter (orange background).
• Figure 3: Fairness iGARBE distributions for both FRS - The estimations were performed using a KDE with Gaussian kernels with a bandwidth determined using Scott's rule. These distributions are the basis for CoFair.
• Figure 4: Relative frequency of occurrence of attribute-label pairs after filtering for attribute combination achieving highest iGARBE score - The filter-dictating combination is chosen based on Tables \ref{['tab:combined-arcface']} and \ref{['tab:combined-facenet']} for ArcFace and FaceNet, respectively. The shown distributions are computed over those annotated samples in the used database that remain after filtering. Attributes part of the filter-dictating combination are not displayed. The distributions help to understand what gender-fair data might look like.