Lights, Camera, Matching: The Role of Image Illumination in Fair Face Recognition

TL;DR

This work investigates how image illumination, specifically face-region brightness, drives demographic disparities in face recognition accuracy between Caucasian and African American females. It introduces three illumination-balancing approaches—Brightness Value Difference (BVD), Brightness Distribution Modality (BDM), and Brightness Distribution IoU (BD-IoU)—evaluated on the MORPH mugshot dataset with an ArcFace-based matcher. All three strategies reduce the baseline $d'$ gap between CF and AF mated-score distributions and improve mean recognition scores for both groups, with substantial reductions observed in several configurations. The results offer actionable guidance for acquisition settings and data collection to achieve fairer performance across demographics, particularly in controlled environments like licenses or passports.

Abstract

Facial brightness is a key image quality factor impacting face recognition accuracy differentials across demographic groups. In this work, we aim to decrease the accuracy gap between the similarity score distributions for Caucasian and African American female mated image pairs, as measured by d' between distributions. To balance brightness across demographic groups, we conduct three experiments, interpreting brightness in the face skin region either as median pixel value or as the distribution of pixel values. Balancing based on median brightness alone yields up to a 46.8% decrease in d', while balancing based on brightness distribution yields up to a 57.6% decrease. In all three cases, the similarity scores of the individual distributions improve, with mean scores maximally improving 5.9% for Caucasian females and 3.7% for African American females.

Figures (11)

• Figure 1: Baseline distributions and $\bar{x}_u$ values for CF / AF. The distribution of CF mated similarity scores is shifted toward lower similarity, relative to the AF distribution.
• Figure 2: Data pre-processing pipeline.
• Figure 3: Ex. images with min., avg., and max. BVs.
• Figure 4: Example pairs with the min, avg., and max BVDs.
• Figure 5: Relationship between mean BVD ($\bar{x}_b$) decrease and d' decrease.