Mitigating Bias in Facial Recognition Systems: Centroid Fairness Loss Optimization

TL;DR

This work tackles bias in facial recognition by introducing a post-processing Centroid Fairness framework that aligns subgroup ROC behavior through pseudo-scores derived from class centroids. A lightweight Fairness Module transforms embeddings and learns updated centroids to produce scores whose FAR/FRR curves match a chosen reference group, via score transformations that quantify alignment. The Centroid Fairness Loss jointly optimizes FAR and FRR regression objectives with a carefully designed weighting scheme, enabling fairness improvements without retraining large FR models. Empirical results on diverse pre-trained models and datasets (e.g., ARCFace variants on RFW and FairFace) show meaningful reductions in bias (BFAR/BFRR) while maintaining competitive accuracy, supporting robust post-processing debiasing across architectures and data shifts.

Abstract

The urging societal demand for fair AI systems has put pressure on the research community to develop predictive models that are not only globally accurate but also meet new fairness criteria, reflecting the lack of disparate mistreatment with respect to sensitive attributes ($\textit{e.g.}$ gender, ethnicity, age). In particular, the variability of the errors made by certain Facial Recognition (FR) systems across specific segments of the population compromises the deployment of the latter, and was judged unacceptable by regulatory authorities. Designing fair FR systems is a very challenging problem, mainly due to the complex and functional nature of the performance measure used in this domain ($\textit{i.e.}$ ROC curves) and because of the huge heterogeneity of the face image datasets usually available for training. In this paper, we propose a novel post-processing approach to improve the fairness of pre-trained FR models by optimizing a regression loss which acts on centroid-based scores. Beyond the computational advantages of the method, we present numerical experiments providing strong empirical evidence of the gain in fairness and of the ability to preserve global accuracy.

Figures (6)

• Figure 1: Pseudo-score transformation to achieve fairness. From a pseudo-score $\overline{s}_a^{(-)}$ (resp. $\overline{s}_a^{(+)}$) of an image-centroid impostor (resp. genuine) pair sharing the attribute $a$, the pseudo-metric $\overline{\mathrm{FAR}}_a(\overline{s}_a^{(-)}) = \alpha$ (resp. $\overline{\mathrm{FRR}}_a(\overline{s}_a^{(+)})=\alpha$) is computed. The transformed score is the score which makes the reference pseudo-metric $\overline{\mathrm{FAR}}_r$ (resp. $\overline{\mathrm{FRR}}_r$) equal to $\alpha$, among the scores from image-centroid pairs of attribute $r$.
• Figure 2: The proposed Fairness Module framework. A frozen pre-trained model $f$ outputs the embedding $f({\bm{x}}_i)$ for the image ${\bm{x}}_i$. The Fairness Module outputs a new fair embedding $g_\theta({\bm{x}}_i)$.
• Figure 3: Pseudo-metrics $\overline{\mathrm{FRR}}_a$ obtained with pseudo-scores, for each race. The pseudo-scores come from the pre-trained model $f$ (solid lines) or the Fairness Module (dashed).
• Figure 4: Real metrics $\mathrm{FRR}_a$ obtained with real scores, for each race. The real scores come from the pre-trained model $f$ (solid lines) or the Fairness Module (dashed).
• Figure 5: Pseudo-metrics $\overline{\mathrm{FAR}}_a$ obtained with pseudo-scores, for each race. The pseudo-scores come either from the pre-trained model $f$ (solid lines), or from the Fairness Module (dashed lines).

Theorems & Definitions (1)

• Proposition B.1