Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Tails when Comparing Distributions: Comprehensive Equity Index (CEI) with Application to Bias Evaluation in Operational Face Biometrics

Imanol Solano, Julian Fierrez, Aythami Morales, Alejandro Peña, Ruben Tolosana, Francisco Zamora-Martinez, Javier San Agustin

TL;DR

This work tackles bias in high-performance face recognition by introducing the Comprehensive Equity Index (CEI), a tail-aware, threshold-agnostic metric that analyzes genuine and impostor score distributions separately. CEI splits each distribution at a percentile to emphasize distribution tails and uses KL divergence to compare to a mean distribution, with CEIN and CEIE variants; an automated CEIA version further selects splitting percentiles and tail weights. Through synthetic and real-world experiments across diverse datasets and biased training conditions, CEI and CEIA demonstrate superior sensitivity to subtle demographic biases in both tail regions and overall shapes, outperforming prior distribution-based metrics. The approach provides a practically useful, adaptable fairness tool that complements traditional FR metrics and can be extended to other domains involving tail analysis of statistical distributions.

Abstract

Demographic bias in high-performance face recognition (FR) systems often eludes detection by existing metrics, especially with respect to subtle disparities in the tails of the score distribution. We introduce the Comprehensive Equity Index (CEI), a novel metric designed to address this limitation. CEI uniquely analyzes genuine and impostor score distributions separately, enabling a configurable focus on tail probabilities while also considering overall distribution shapes. Our extensive experiments (evaluating state-of-the-art FR systems, intentionally biased models, and diverse datasets) confirm CEI's superior ability to detect nuanced biases where previous methods fall short. Furthermore, we present CEI^A, an automated version of the metric that enhances objectivity and simplifies practical application. CEI provides a robust and sensitive tool for operational FR fairness assessment. The proposed methods have been developed particularly for bias evaluation in face biometrics but, in general, they are applicable for comparing statistical distributions in any problem where one is interested in analyzing the distribution tails.

Balancing Tails when Comparing Distributions: Comprehensive Equity Index (CEI) with Application to Bias Evaluation in Operational Face Biometrics

TL;DR

This work tackles bias in high-performance face recognition by introducing the Comprehensive Equity Index (CEI), a tail-aware, threshold-agnostic metric that analyzes genuine and impostor score distributions separately. CEI splits each distribution at a percentile to emphasize distribution tails and uses KL divergence to compare to a mean distribution, with CEIN and CEIE variants; an automated CEIA version further selects splitting percentiles and tail weights. Through synthetic and real-world experiments across diverse datasets and biased training conditions, CEI and CEIA demonstrate superior sensitivity to subtle demographic biases in both tail regions and overall shapes, outperforming prior distribution-based metrics. The approach provides a practically useful, adaptable fairness tool that complements traditional FR metrics and can be extended to other domains involving tail analysis of statistical distributions.

Abstract

Demographic bias in high-performance face recognition (FR) systems often eludes detection by existing metrics, especially with respect to subtle disparities in the tails of the score distribution. We introduce the Comprehensive Equity Index (CEI), a novel metric designed to address this limitation. CEI uniquely analyzes genuine and impostor score distributions separately, enabling a configurable focus on tail probabilities while also considering overall distribution shapes. Our extensive experiments (evaluating state-of-the-art FR systems, intentionally biased models, and diverse datasets) confirm CEI's superior ability to detect nuanced biases where previous methods fall short. Furthermore, we present CEI^A, an automated version of the metric that enhances objectivity and simplifies practical application. CEI provides a robust and sensitive tool for operational FR fairness assessment. The proposed methods have been developed particularly for bias evaluation in face biometrics but, in general, they are applicable for comparing statistical distributions in any problem where one is interested in analyzing the distribution tails.

Paper Structure

This paper contains 22 sections, 14 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Methodology: Fairness in Operational Biometric Systems
  4. Existing Fairness Metrics
  5. Differential Outcome Metrics
  6. Differential Performance Metrics
  7. Proposed Fairness Metric: Comprehensive Equity Index (CEI)
  8. Automated Parameter Selection for CEI (CEIA)
  9. Experimental Framework
  10. Datasets
  11. Training Datasets.
  12. Evaluation Datasets.
  13. Face Recognition Models
  14. Experiments: Results and Interpretation
  15. Synthetically-Generated Distributions
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Genuine and impostor synthetically-generated similarity score distributions, in different scenarios: (Left) Biased Genuine distribution tail (BG); (Center) Biased Impostor distribution tail (BI); and (Right) Biased genuine-impostor distribution Center (BC).
  • Figure 2: Genuine (continuous line) and impostor (dashed line) distributions for ResNet-100 He2015DeepRL model in MORPH Ricanek2006MORPH (Left) and RFW Wang2018RacialFI (Right) datasets. The x-axis shows the cosine similarity between two images. Thus the impostor distributions are on the left and the genuine on the right. Each demographic group is represented by a different color.