Mitigating Gender Bias in Face Recognition Using the von Mises-Fisher Mixture Model

TL;DR

This work tackles gender bias in face recognition by introducing two deployment-focused fairness metrics, BFAR and BFRR, and a post-processing solution that leverages a Fair von Mises-Fisher loss within a shallow Ethical Module. The method models embeddings on a hypersphere as a vMF mixture with two gender-dependent concentration parameters, enabling targeted adjustment of intra-class variance to reduce bias while preserving accuracy. Empirical results on IJB-C and LFW demonstrate meaningful reductions in bias metrics with competitive or minimal sacrifices in FR performance, and show robustness across architectures and backbones. Overall, the approach offers a simple, fast, and effective post-processing route to fairness in FR, with potential applicability to other bias types and settings.

Abstract

In spite of the high performance and reliability of deep learning algorithms in a wide range of everyday applications, many investigations tend to show that a lot of models exhibit biases, discriminating against specific subgroups of the population (e.g. gender, ethnicity). This urges the practitioner to develop fair systems with a uniform/comparable performance across sensitive groups. In this work, we investigate the gender bias of deep Face Recognition networks. In order to measure this bias, we introduce two new metrics, $\mathrm{BFAR}$ and $\mathrm{BFRR}$, that better reflect the inherent deployment needs of Face Recognition systems. Motivated by geometric considerations, we mitigate gender bias through a new post-processing methodology which transforms the deep embeddings of a pre-trained model to give more representation power to discriminated subgroups. It consists in training a shallow neural network by minimizing a Fair von Mises-Fisher loss whose hyperparameters account for the intra-class variance of each gender. Interestingly, we empirically observe that these hyperparameters are correlated with our fairness metrics. In fact, extensive numerical experiments on a variety of datasets show that a careful selection significantly reduces gender bias. The code used for the experiments can be found at https://github.com/JRConti/EthicalModule_vMF.

Figures (15)

• Figure 1: Illustration of the Ethical Module methodology. In gray: our experiment choices.
• Figure 2: Illustration of the geometric nature of bias. Each point is the embedding of an image. In green: two male identities. In red: two female identities. The overlapping region between two identities is higher for females than for males. The grey circles are the acceptance zones, centered around an embedding of reference, associated to a constant threshold $t$ of acceptance.
• Figure 3: $500$ samples from the vMF distribution in dimension $3$ with parameters ${\bm{\mu}} =$ [$0.5$, $0$, $\sqrt{0.75}$] and $\kappa > 0$.
• Figure 4: Illustration of a vMF mixture model.
• Figure 5: Fairness and evaluation metrics on IJB-C for the Ethical Module when one of the two hyperparameters is fixed. The FAR level defining the threshold $t$ is set to $10^{-3}$; the pre-trained model is ArcFace with a ResNet100 backbone. $\mathrm{FRR}@\mathrm{FAR}$ is expressed as a percentage (%). The three versions of the Ethical Module presented in \ref{['tab:kappa_choice']} are annotated with circles.