Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fairness Under Cover: Evaluating the Impact of Occlusions on Demographic Bias in Facial Recognition

Rafael M. Mamede, Pedro C. Neto, Ana F. Sequeira

TL;DR

The paper addresses fairness in occluded face recognition by introducing the Face Occlusion Impact Ratio (FOIR) to quantify how occluded regions interact with model reliance on different demographic groups. It analyzes occluded (RFW1, RFW4) and unoccluded (RFW0) conditions using two architectures (ResNet34, ResNet50) trained with ElasticArcFace on two datasets (BUPT-Balanced, BUPT-GlobalFace) and evaluates a suite of group fairness metrics. The results show that occlusions worsen overall performance and amplify demographic disparities, with African individuals typically experiencing larger accuracy drops, and that some ratio-based metrics may falsely appear more fair due to higher overall error; FOIR and pixel-attribution analyses reveal ethnicity-specific occlusion effects in false non-match cases. The work highlights the need to assess fairness under realistic occlusions and provides FOIR as a practical tool for diagnosing and mitigating bias in occluded face recognition.

Abstract

This study investigates the effects of occlusions on the fairness of face recognition systems, particularly focusing on demographic biases. Using the Racial Faces in the Wild (RFW) dataset and synthetically added realistic occlusions, we evaluate their effect on the performance of face recognition models trained on the BUPT-Balanced and BUPT-GlobalFace datasets. We note increases in the dispersion of FMR, FNMR, and accuracy alongside decreases in fairness according to Equilized Odds, Demographic Parity, STD of Accuracy, and Fairness Discrepancy Rate. Additionally, we utilize a pixel attribution method to understand the importance of occlusions in model predictions, proposing a new metric, Face Occlusion Impact Ratio (FOIR), that quantifies the extent to which occlusions affect model performance across different demographic groups. Our results indicate that occlusions exacerbate existing demographic biases, with models placing higher importance on occlusions in an unequal fashion, particularly affecting African individuals more severely.

Fairness Under Cover: Evaluating the Impact of Occlusions on Demographic Bias in Facial Recognition

TL;DR

The paper addresses fairness in occluded face recognition by introducing the Face Occlusion Impact Ratio (FOIR) to quantify how occluded regions interact with model reliance on different demographic groups. It analyzes occluded (RFW1, RFW4) and unoccluded (RFW0) conditions using two architectures (ResNet34, ResNet50) trained with ElasticArcFace on two datasets (BUPT-Balanced, BUPT-GlobalFace) and evaluates a suite of group fairness metrics. The results show that occlusions worsen overall performance and amplify demographic disparities, with African individuals typically experiencing larger accuracy drops, and that some ratio-based metrics may falsely appear more fair due to higher overall error; FOIR and pixel-attribution analyses reveal ethnicity-specific occlusion effects in false non-match cases. The work highlights the need to assess fairness under realistic occlusions and provides FOIR as a practical tool for diagnosing and mitigating bias in occluded face recognition.

Abstract

This study investigates the effects of occlusions on the fairness of face recognition systems, particularly focusing on demographic biases. Using the Racial Faces in the Wild (RFW) dataset and synthetically added realistic occlusions, we evaluate their effect on the performance of face recognition models trained on the BUPT-Balanced and BUPT-GlobalFace datasets. We note increases in the dispersion of FMR, FNMR, and accuracy alongside decreases in fairness according to Equilized Odds, Demographic Parity, STD of Accuracy, and Fairness Discrepancy Rate. Additionally, we utilize a pixel attribution method to understand the importance of occlusions in model predictions, proposing a new metric, Face Occlusion Impact Ratio (FOIR), that quantifies the extent to which occlusions affect model performance across different demographic groups. Our results indicate that occlusions exacerbate existing demographic biases, with models placing higher importance on occlusions in an unequal fashion, particularly affecting African individuals more severely.
Paper Structure (13 sections, 17 equations, 2 figures, 4 tables)

This paper contains 13 sections, 17 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Experimental Design
  5. Models
  6. Group Fairness Metrics
  7. Experimental Results
  8. Assessing Model Performance and Fairness
  9. Fairness Increase of Ratio-based Metrics on the Occluded Scenarios
  10. Pixel Attribution in the Occluded Scenarios
  11. Conclusion
  12. Acknowledgments
  13. Accuracies, FMR and FNMR

Figures (2)

  • Figure 1: Genuine pair with probe image occluded, leading to incorrect classification by the Balanced34 model. Note the difference in the importance maps obtained with xSSAB huber2024efficient: when the image is occluded most of the important pixels fall on the occluded regions. (Best viewed in color)
  • Figure 2: Examples of occlusions added with protocol 1 and protocol 4 (introduced in the 2022 Competition on Occluded Face Recognition neto2022ocfr) to the RFW dataset.