Explainable Face Recognition via Improved Localization

TL;DR

The paper tackles explainability in deep learning-based face recognition by requiring visual explanations of decisions. It introduces Scaled Directed Divergence (SDD), a CAM-based discriminative localization technique that produces narrow, class-specific explanations by scaling the target CAM and diverging from others, with the CAM for class $c$ given by $M_c(x,y)=\sum_k w_k^c f_k(x,y)$ and an amplification factor $\alpha$ defined as $ \alpha = 0.2 + \frac{\mathrm{stddev}(CAM_{SDD})}{\mathrm{stddev}(CAM_{SDD})+1}\cdot 0.2$. The AdaFace model (ResNet-100) pretrained on WebFace12M and finetuned on FaceScrub achieves a 96.33% test accuracy on a 80/10/10 split, enabling robust evaluation of explanations. Deletion and retention analyses show that SDD CAM highlights more faithful and narrowly localized regions than random CAMs, supporting greater transparency and the potential for bias detection and model improvement.

Abstract

Biometric authentication has become one of the most widely used tools in the current technological era to authenticate users and to distinguish between genuine users and imposters. Face is the most common form of biometric modality that has proven effective. Deep learning-based face recognition systems are now commonly used across different domains. However, these systems usually operate like black-box models that do not provide necessary explanations or justifications for their decisions. This is a major disadvantage because users cannot trust such artificial intelligence-based biometric systems and may not feel comfortable using them when clear explanations or justifications are not provided. This paper addresses this problem by applying an efficient method for explainable face recognition systems. We use a Class Activation Mapping (CAM)-based discriminative localization (very narrow/specific localization) technique called Scaled Directed Divergence (SDD) to visually explain the results of deep learning-based face recognition systems. We perform fine localization of the face features relevant to the deep learning model for its prediction/decision. Our experiments show that the SDD Class Activation Map (CAM) highlights the relevant face features very specifically compared to the traditional CAM and very accurately. The provided visual explanations with narrow localization of relevant features can ensure much-needed transparency and trust for deep learning-based face recognition systems.

Figures (6)

• Figure 1: Steps of generating CAMs for visual explainability R4. The most important layers in CNN are convolutional layers, which capture significant features of the input image. The feature maps of the last convolutional layer are global average pooled before the output layer. The weights of the output layer are mapped back to the convolutional feature maps (last convolutional layer) to generate the CAM for the input class. The CAM is a weighted sum of the convolutional feature maps.
• Figure 2: An overview of the Scaled Directed Divergence (SDD) technique R6. For an input image, multiple CAMs are generated for multiple classes that are overlaid on the same test image. Among the classes for which CAMs are generated, one class is selected as the target class (the selection of the target class is up to the user; it can be the original class, the predicted class, or any other class). Here, we consider class_1 as the target class. As I mentioned, the target class is not fixed, and any class can be considered the target class. The target class CAM is multiplied by the scaling factor (SF) to adjust for differences in magnitude between CAMs to enable meaningful subtraction. Other CAMs are subtracted from the scaled target class CAM to generate the Divergence values. The Divergence values are multiplied by alpha (explained in section \ref{['SDD']}) for amplification/decrease before exponentiation. Finally, the SDD CAM is generated for the target class.
• Figure 3: Visually explaining the prediction of the face recognition model (described in section \ref{['model-1']}) by highlighting the relevant face features. The class of the test image is subject_4. The DL model made a correct prediction of the test image. The probability of the top 5 classes predicted by the model: subject_4 (99.025%), subject_333 (.118%), subject_482 (.103%), subject_475 (.089%), subject_518 (.087%). In a CAM, the red, yellow, and green regions highlight the relevant features while the blue regions highlight the non-relevant features. The top row shows the example images of the classes. The target class is subject_4. In the bottom row, the first image (from the left) shows the traditional CAM for class subject_4. The last image (SDD_subject_4) shows the SDD CAM for target class subject_4, which localizes the most relevant features (left side area beside the nose and middle of the forehead) in a very narrow manner (compared to the traditional CAM). Other CAMs are subtracted from the scaled target class CAM to generate the Divergence values, which are amplified/decreased and exponentiated to generate the SDD CAM.
• Figure 4: Visually explaining the prediction of the face recognition model (described in section \ref{['model-1']}) by highlighting the relevant face features. The class of the test image is subject_120. The DL model made a correct prediction of the test image. The target class is subject_120. The probability of the top 5 classes predicted by the model: subject_120 (99.817%), subject_398 (.029%), subject_285 (.022%), subject_490 (.011%), subject_303 (.009%). The top row shows the example images of the classes. In the bottom row, the first image (from the left) shows the traditional CAM for class subject_120. The last image (SDD_subject_120) shows the SDD CAM for target class subject_120, which localizes the most relevant features (right side area beside the lips) in a very narrow manner (compared to the traditional CAM). Other CAMs are subtracted from the scaled target class CAM to generate the Divergence values, which are amplified/decreased and exponentiated to generate the SDD CAM.
• Figure 5: An overview of the deletion scheme, which is used for the evaluation of the visual explanation generated by the SDD method. From the left, the first image is the original image, the second image shows the SDD CAM (SDD CAM is overlaid on the image), the top 20% SDD CAM regions are deleted in the third image, and random CAM regions of the same dimension are deleted in the fourth image.