Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VerLM: Explaining Face Verification Using Natural Language

Syed Abdul Hannan, Hazim Bukhari, Thomas Cantalapiedra, Eman Ansar, Massa Baali, Rita Singh, Bhiksha Raj

TL;DR

VerLM tackles the lack of transparency in face verification by jointly performing verification and generating natural-language explanations grounded in paired facial embeddings. It adapts an audio-diff cross-projection paradigm to the visual domain, introducing a six-component architecture (including a cross-projection layer and separator token) and a three-stage training schedule with a Diversity Loss to produce concise and comprehensive explanations. A new Explainable Face Verification Dataset (built from VGGFace2) underpins evaluation, and extensive ablations show that cross-projection, textual conditioning, model scale, and staged training yield superior, semantically aligned explanations compared to baselines. The approach advances trustworthy face recognition by combining high verification accuracy with human-interpretable rationales, with potential impact on bias detection, accountability, and user trust in deployed systems.

Abstract

Face verification systems have seen substantial advancements; however, they often lack transparency in their decision-making processes. In this paper, we introduce an innovative Vision-Language Model (VLM) for Face Verification, which not only accurately determines if two face images depict the same individual but also explicitly explains the rationale behind its decisions. Our model is uniquely trained using two complementary explanation styles: (1) concise explanations that summarize the key factors influencing its decision, and (2) comprehensive explanations detailing the specific differences observed between the images. We adapt and enhance a state-of-the-art modeling approach originally designed for audio-based differentiation to suit visual inputs effectively. This cross-modal transfer significantly improves our model's accuracy and interpretability. The proposed VLM integrates sophisticated feature extraction techniques with advanced reasoning capabilities, enabling clear articulation of its verification process. Our approach demonstrates superior performance, surpassing baseline methods and existing models. These findings highlight the immense potential of vision language models in face verification set up, contributing to more transparent, reliable, and explainable face verification systems.

VerLM: Explaining Face Verification Using Natural Language

TL;DR

VerLM tackles the lack of transparency in face verification by jointly performing verification and generating natural-language explanations grounded in paired facial embeddings. It adapts an audio-diff cross-projection paradigm to the visual domain, introducing a six-component architecture (including a cross-projection layer and separator token) and a three-stage training schedule with a Diversity Loss to produce concise and comprehensive explanations. A new Explainable Face Verification Dataset (built from VGGFace2) underpins evaluation, and extensive ablations show that cross-projection, textual conditioning, model scale, and staged training yield superior, semantically aligned explanations compared to baselines. The approach advances trustworthy face recognition by combining high verification accuracy with human-interpretable rationales, with potential impact on bias detection, accountability, and user trust in deployed systems.

Abstract

Face verification systems have seen substantial advancements; however, they often lack transparency in their decision-making processes. In this paper, we introduce an innovative Vision-Language Model (VLM) for Face Verification, which not only accurately determines if two face images depict the same individual but also explicitly explains the rationale behind its decisions. Our model is uniquely trained using two complementary explanation styles: (1) concise explanations that summarize the key factors influencing its decision, and (2) comprehensive explanations detailing the specific differences observed between the images. We adapt and enhance a state-of-the-art modeling approach originally designed for audio-based differentiation to suit visual inputs effectively. This cross-modal transfer significantly improves our model's accuracy and interpretability. The proposed VLM integrates sophisticated feature extraction techniques with advanced reasoning capabilities, enabling clear articulation of its verification process. Our approach demonstrates superior performance, surpassing baseline methods and existing models. These findings highlight the immense potential of vision language models in face verification set up, contributing to more transparent, reliable, and explainable face verification systems.
Paper Structure (17 sections, 2 equations, 1 figure, 8 tables)

This paper contains 17 sections, 2 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Explainable Face Verification
  3. Explainable Face Verification Dataset
  4. Model
  5. Training
  6. Results
  7. Experimental Setup
  8. Evaluation Metrics and Performance Analysis
  9. Ablations
  10. Training Strategy
  11. Scaling of Language Model
  12. Model Strategy
  13. Textual Encoder Strategy
  14. Different Image Encoders
  15. Model Scaling
  16. ...and 2 more sections

Figures (1)

  • Figure 1: VerLM takes in two images and text prompt as inputs to the model. It processes the images through an image encoder and then through the image projection layer. The text is put through a text embedder followed by a text projection layer. The two encoded images are concatenated with an separator token in between and then further concatenated with the text projection layer output. This is then put through a cross-projection layer. Finally, the output of the cross-projection layer is input to the decoder model which autoregressively generates the description