FairDeFace: Evaluating the Fairness and Adversarial Robustness of Face Obfuscation Methods

TL;DR

FairDeFace introduces a comprehensive, open-source framework to evaluate both fairness and adversarial robustness of face obfuscation methods across diverse datasets and threat models. It combines obfuscated-face generation, detection/recognition, privacy and utility assessments, and bias visualization using saliency maps and MediaPipe facial landmarks, enabling cross-method comparisons with standardized metrics. Across seven obfuscation methods and five datasets, GAN-based approaches like DeepPrivacy2 show strong privacy and relatively low bias, while methods such as Fawkes and CIAGAN reveal notable demographic biases and limited robustness. The work emphasizes that detector quality and dataset characteristics influence fairness assessments and advocates holistic, demographic-aware obfuscation strategies augmented by feature-focus analyses. Overall, FairDeFace provides a rigorous toolset for researchers and practitioners to benchmark, understand, and improve fairness in privacy-preserving facial obfuscation.

Abstract

The lack of a common platform and benchmark datasets for evaluating face obfuscation methods has been a challenge, with every method being tested using arbitrary experiments, datasets, and metrics. While prior work has demonstrated that face recognition systems exhibit bias against some demographic groups, there exists a substantial gap in our understanding regarding the fairness of face obfuscation methods. Providing fair face obfuscation methods can ensure equitable protection across diverse demographic groups, especially since they can be used to preserve the privacy of vulnerable populations. To address these gaps, this paper introduces a comprehensive framework, named FairDeFace, designed to assess the adversarial robustness and fairness of face obfuscation methods. The framework introduces a set of modules encompassing data benchmarks, face detection and recognition algorithms, adversarial models, utility detection models, and fairness metrics. FairDeFace serves as a versatile platform where any face obfuscation method can be integrated, allowing for rigorous testing and comparison with other state-of-the-art methods. In its current implementation, FairDeFace incorporates 6 attacks, and several privacy, utility and fairness metrics. Using FairDeFace, and by conducting more than 500 experiments, we evaluated and compared the adversarial robustness of seven face obfuscation methods. This extensive analysis led to many interesting findings both in terms of the degree of robustness of existing methods and their biases against some gender or racial groups. FairDeFace also uses visualization of focused areas for both obfuscation and verification attacks to show not only which areas are mostly changed in the obfuscation process for some demographics, but also why they failed through focus area comparison of obfuscation and verification.

Figures (7)

• Figure 1: FairDeFace Framework
• Figure 2: Obfuscated faces for a face example
• Figure 3: Visualization of Bias Overview
• Figure 4: Visualization of the Fawkes's Process. Left: Original image. Center: Fawkes-protected image. Right: Saliency map heatmap showing obfuscation focus areas.
• Figure 5: Focus Features Distribution Across Races (Combined Genders) for Fawkes