FineFACE: Fair Facial Attribute Classification Leveraging Fine-grained Features

TL;DR

This paper proposes a novel approach to fair facial attribute classification by framing it as a fine-grained classification problem that obtains a Pareto-efficient balance between accuracy and fairness between demographic groups.

Abstract

Published research highlights the presence of demographic bias in automated facial attribute classification algorithms, particularly impacting women and individuals with darker skin tones. Existing bias mitigation techniques typically require demographic annotations and often obtain a trade-off between fairness and accuracy, i.e., Pareto inefficiency. Facial attributes, whether common ones like gender or others such as "chubby" or "high cheekbones", exhibit high interclass similarity and intraclass variation across demographics leading to unequal accuracy. This requires the use of local and subtle cues using fine-grained analysis for differentiation. This paper proposes a novel approach to fair facial attribute classification by framing it as a fine-grained classification problem. Our approach effectively integrates both low-level local features (like edges and color) and high-level semantic features (like shapes and structures) through cross-layer mutual attention learning. Here, shallow to deep CNN layers function as experts, offering category predictions and attention regions. An exhaustive evaluation on facial attribute annotated datasets demonstrates that our FineFACE model improves accuracy by 1.32% to 1.74% and fairness by 67% to 83.6%, over the SOTA bias mitigation techniques. Importantly, our approach obtains a Pareto-efficient balance between accuracy and fairness between demographic groups. In addition, our approach does not require demographic annotations and is applicable to diverse downstream classification tasks. To facilitate reproducibility, the code and dataset information is available at https://github.com/VCBSL-Fairness/FineFACE.

Figures (4)

• Figure 1: Visualization of the attention map obtained by our proposed FineFACE over baseline (both using ResNet50 backbone) for facial attribute classification. The highly activated region is shown by the red zone on the map, followed by yellow, green, and blue zones. Top: "High Cheekbones" classifier. Bottom: "Smiling" classifier.
• Figure 2: FineFACE network structure. This figure illustrates this method by introducing three experts $e_{1}$, $e_{2}$, $e_{3}$, on a 5-stage backbone CNN (e.g., ResNet50). The working of each expert and the concatenation of experts are depicted in different colors. Each expert receives feature maps from a specific layer as input and generates a categorical prediction along with an attention region, which is used for data augmentation by other experts. This architecture is trained in multiple steps within each iteration. We start by training the deepest expert (e3), followed by the shallower experts. Finally, in the last step, we train the concatenation of experts to enhance overall performance.
• Figure 3: Visualization Results of Gender Classifier. Left through right in each set of images are the input image from FairFace dataset, visualization results based on our FineFACE method's 3 experts ($\tilde{\Omega}_{1}^{norm}$, $\tilde{\Omega}_{2}^{norm}$, $\tilde{\Omega}_{3}^{norm}$), and our method's final visualization ($\tilde{\Omega}_{oval}^{norm}$), versus a basic ResNet50 architecture's final visualization ($\tilde{\Omega}_{ori}^{norm}$). Our FineFACE captures a more comprehensive feature representation of the image, thereby enhancing fairness as well as accuracy.
• Figure 4: Fairness methods on the CelebA dataset. We report mean scores over 13 gender independent target attributes. Left: Change in minimum group accuracy difference relative to the single task baseline classifier. Top Right: A decomposition of the change in accuracy from baseline into overall change, change in the best performing group, and the worst. Center Right: Decomposition of the change in True Positive Rate (TPR). Bottom Right: plots of DEO and DEOdds.