LabellessFace: Fair Metric Learning for Face Recognition without Attribute Labels

TL;DR

The paper tackles demographic bias in face recognition without requiring demographic labels. It introduces LabellessFace, which uses a class favoritism level to govern a per-class margin in a softmax-based metric learning framework. Key contributions are (i) the class favoritism level, (ii) the fair class margin penalty with per-class margin coefficients defined as $d_c = 2/(1+exp(\gamma f_c))$ for $f_c<0$, and $d_c = 2/(1+exp(\gamma h f_c))$ for $f_c \ge 0$, and (iii) an epoch-end update of $f_c$ guiding $d_c$. Experiments on the BUPT-Balancedface, RFW, and LFW datasets show improved fairness metrics such as $STD$, $Gini$, and $SER$ while maintaining authentication accuracy, indicating scalability to unknown attributes.

Abstract

Demographic bias is one of the major challenges for face recognition systems. The majority of existing studies on demographic biases are heavily dependent on specific demographic groups or demographic classifier, making it difficult to address performance for unrecognised groups. This paper introduces ``LabellessFace'', a novel framework that improves demographic bias in face recognition without requiring demographic group labeling typically required for fairness considerations. We propose a novel fairness enhancement metric called the class favoritism level, which assesses the extent of favoritism towards specific classes across the dataset. Leveraging this metric, we introduce the fair class margin penalty, an extension of existing margin-based metric learning. This method dynamically adjusts learning parameters based on class favoritism levels, promoting fairness across all attributes. By treating each class as an individual in facial recognition systems, we facilitate learning that minimizes biases in authentication accuracy among individuals. Comprehensive experiments have demonstrated that our proposed method is effective for enhancing fairness while maintaining authentication accuracy.

Figures (4)

• Figure 1: Overview of the LabellessFace framework.
• Figure 2: Overview of the Class Favoritism Level calculation.
• Figure 3: The gradient change of the margin coefficient $d_c$ with respect to the value of the harmony coefficient.
• Figure 4: The fairness heatmap of each model across 26 attributes on LFW: Each cell indicates the deviation of EER, with blue indicating lower EER than the average and red indicating higher EER than the average. For reference, we include the values of the standard deviation (STD) from Table \ref{['table:fairness']} in parentheses.