Bridging Human Concepts and Computer Vision for Explainable Face Verification

TL;DR

This paper uses Mediapipe, which provides a segmentation technique that identifies distinct human-semantic facial regions, enabling the machine's perception analysis, and adapted two model-agnostic algorithms to provide human-interpretable insights into the decision-making processes.

Abstract

With Artificial Intelligence (AI) influencing the decision-making process of sensitive applications such as Face Verification, it is fundamental to ensure the transparency, fairness, and accountability of decisions. Although Explainable Artificial Intelligence (XAI) techniques exist to clarify AI decisions, it is equally important to provide interpretability of these decisions to humans. In this paper, we present an approach to combine computer and human vision to increase the explanation's interpretability of a face verification algorithm. In particular, we are inspired by the human perceptual process to understand how machines perceive face's human-semantic areas during face comparison tasks. We use Mediapipe, which provides a segmentation technique that identifies distinct human-semantic facial regions, enabling the machine's perception analysis. Additionally, we adapted two model-agnostic algorithms to provide human-interpretable insights into the decision-making processes.

Figures (11)

• Figure 1: Face verification adaptation of XAI Perceptual processing framework proposed by 34zhang2022towards and inspired by how humans process stimuli (select, organize, interpret and compare)
• Figure 2: Proposed flowchart. We extract concepts from the face verification model (using KernelSHAP) and input them into a Semantic Face perturbation phase. In this phase, the two images' perturbation is made in the same regions to evaluate similarities and dissimilarities. We propose three algorithms for the perturbations: Single removal, greedy removal, and average similarity map.
• Figure 3: An interpretation of a relational/configural model of face perception.
• Figure 4: In the image (a) Mediapipe landmarks are plotted on the sample image. In the image (b), the 13 semantic sections are defined through the landmarks
• Figure 5: Examples of two images' human-semantics part importance scores using KernelSHAP lundberg2017nips. We analyse two models: CasiaNet yi2014learning in (a) and (c), and VGGfaces2 massoli2020ivc in (b) and (d). Green parts are more important according to Shap scores. There are differences between important parts for different images, especially for VGGfaces2. That is why we aggregate ranked importance over 200 images.