Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reliable and Reproducible Demographic Inference for Fairness in Face Analysis

Alexandre Fournier-Montgieux, Hervé Le Borgne, Adrian Popescu, Bertrand Luvison

TL;DR

This work tackles the reliability gap in demographic attribute inference (DAI) for fairness auditing in face analysis systems by proposing a modular transfer-learning pipeline that freezes pretrained face recognition backbones and trains non-linear classification heads. The authors introduce a taxonomy-agnostic robustness metric, HomE, to capture intra-identity label consistency and provide a theoretical framework showing that improved DAI accuracy reduces bias and variance in downstream fairness estimates. Empirical results across ethnicity and gender inference show substantial improvements in accuracy, fairness, and robustness, with ethnicity benefiting most from the approach, and larger backbones providing additional gains. The study emphasizes transparency and reproducibility by committing to public release of data processing scripts, models, and evaluation tools, and discusses ethical considerations, limitations, and future directions for finer-grained segmentation and interpretability. Overall, the paper establishes a practical, scalable foundation for reliable fairness auditing in face analysis through a reproducible DAI paradigm and rigorous robustness assessment.

Abstract

Fairness evaluation in face analysis systems (FAS) typically depends on automatic demographic attribute inference (DAI), which itself relies on predefined demographic segmentation. However, the validity of fairness auditing hinges on the reliability of the DAI process. We begin by providing a theoretical motivation for this dependency, showing that improved DAI reliability leads to less biased and lower-variance estimates of FAS fairness. To address this, we propose a fully reproducible DAI pipeline that replaces conventional end-to-end training with a modular transfer learning approach. Our design integrates pretrained face recognition encoders with non-linear classification heads. We audit this pipeline across three dimensions: accuracy, fairness, and a newly introduced notion of robustness, defined via intra-identity consistency. The proposed robustness metric is applicable to any demographic segmentation scheme. We benchmark the pipeline on gender and ethnicity inference across multiple datasets and training setups. Our results show that the proposed method outperforms strong baselines, particularly on ethnicity, which is the more challenging attribute. To promote transparency and reproducibility, we will publicly release the training dataset metadata, full codebase, pretrained models, and evaluation toolkit. This work contributes a reliable foundation for demographic inference in fairness auditing.

Reliable and Reproducible Demographic Inference for Fairness in Face Analysis

TL;DR

This work tackles the reliability gap in demographic attribute inference (DAI) for fairness auditing in face analysis systems by proposing a modular transfer-learning pipeline that freezes pretrained face recognition backbones and trains non-linear classification heads. The authors introduce a taxonomy-agnostic robustness metric, HomE, to capture intra-identity label consistency and provide a theoretical framework showing that improved DAI accuracy reduces bias and variance in downstream fairness estimates. Empirical results across ethnicity and gender inference show substantial improvements in accuracy, fairness, and robustness, with ethnicity benefiting most from the approach, and larger backbones providing additional gains. The study emphasizes transparency and reproducibility by committing to public release of data processing scripts, models, and evaluation tools, and discusses ethical considerations, limitations, and future directions for finer-grained segmentation and interpretability. Overall, the paper establishes a practical, scalable foundation for reliable fairness auditing in face analysis through a reproducible DAI paradigm and rigorous robustness assessment.

Abstract

Fairness evaluation in face analysis systems (FAS) typically depends on automatic demographic attribute inference (DAI), which itself relies on predefined demographic segmentation. However, the validity of fairness auditing hinges on the reliability of the DAI process. We begin by providing a theoretical motivation for this dependency, showing that improved DAI reliability leads to less biased and lower-variance estimates of FAS fairness. To address this, we propose a fully reproducible DAI pipeline that replaces conventional end-to-end training with a modular transfer learning approach. Our design integrates pretrained face recognition encoders with non-linear classification heads. We audit this pipeline across three dimensions: accuracy, fairness, and a newly introduced notion of robustness, defined via intra-identity consistency. The proposed robustness metric is applicable to any demographic segmentation scheme. We benchmark the pipeline on gender and ethnicity inference across multiple datasets and training setups. Our results show that the proposed method outperforms strong baselines, particularly on ethnicity, which is the more challenging attribute. To promote transparency and reproducibility, we will publicly release the training dataset metadata, full codebase, pretrained models, and evaluation toolkit. This work contributes a reliable foundation for demographic inference in fairness auditing.
Paper Structure (35 sections, 2 theorems, 36 equations, 4 figures, 4 tables)

This paper contains 35 sections, 2 theorems, 36 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Demographic attribute inference
  4. Theoretical Motivation
  5. Training datasets
  6. Model training
  7. Evaluation dimensions
  8. Experiments
  9. Experimental setup
  10. DAI pipelines.
  11. Test datasets.
  12. Metrics.
  13. Main results
  14. Ethnicity.
  15. Gender.
  16. ...and 20 more sections

Key Result

Proposition 1

(informal) Under the assumptions above, the empirical plug-in estimator $\widehat{m}_g$ converges in probability toward $m_g := \Pr(Y=1\mid \widehat{G} = g)$ as $I\to\infty$ and $m_g=\frac{1}{\tau_g}\sum_{a=1}^K\pi_ac_{ag}p_a$. Moreover, the bias of the plug-in limit relative to the true group rate and therefore if $c_{ag}\to 0$ for every $a\ne g$ and $c_{g g}\to 1$ then $\mathrm{Bias}_g\to 0$. M

Figures (4)

  • Figure 1: Ethnicity inference results (4 classes) audited using accuracy, fairness, and robustness dimensions on the BFW dataset robinson2020face for baselines (FairFace Karkkainen2021FairFace and DeepFace serengil2024benchmark) and several instances of the proposed approach. Higher values are better for accuracy. Lower values are better for fairness. Darker is better for robustness. The size of the circles is proportional to the number of parameters of the backbone DNN used, with smaller circles indicating improved inference efficiency. An ideal DAI system lies at the bottom-right corner with a collapsed circle.
  • Figure 2: Overview of pipelines designed for training and evaluating demographic attribute inference. The raw data labels correspond to a demographic split schema comprising $n$ segments. The model training component implements different strategies. We compare: (1) the baseline learning strategy implemented in FairFaceKarkkainen2021FairFace, which jointly learns the feature extractor $\mathcal{F}$ and classifier $\mathcal{C}$, (2) the learning strategy implemented in DeepFace serengil2024benchmark, which fine-tunes the entire pretrained FR model, and (3) training only the classifier $\mathcal{C}$ on top of the frozen $\mathcal{F}^p$. Finally, we follow common practice to evaluate the accuracy and fairness of the obtained models, and add a robustness evaluation employing a newly introduced metric.
  • Figure 3: Ethnicity segments accuracy in RFW for the two main baselines and IR50+RBF pretrained with 1M WebFace images.
  • Figure 4: Ethnicity segments robustness (lower is better) in RFW for the two main baselines, IR50+RBF pretrained with 1M WebFace images, and with four skin-color-based segments from Thong2023BeyondSkinTone.

Theorems & Definitions (4)

  • Proposition 1
  • Proposition 2
  • proof
  • proof