MLLM-based Textual Explanations for Face Comparison

Abstract

Multimodal Large Language Models (MLLMs) have recently been proposed as a means to generate natural-language explanations for face recognition decisions. While such explanations facilitate human interpretability, their reliability on unconstrained face images remains underexplored. In this work, we systematically analyze MLLM-generated explanations for the unconstrained face verification task on the challenging IJB-S dataset, with a particular focus on extreme pose variation and surveillance imagery. Our results show that even when MLLMs produce correct verification decisions, the accompanying explanations frequently rely on non-verifiable or hallucinated facial attributes that are not supported by visual evidence. We further study the effect of incorporating information from traditional face recognition systems, viz., scores and decisions, alongside the input images. Although such information improves categorical verification performance, it does not consistently lead to faithful explanations. To evaluate the explanations beyond decision accuracy, we introduce a likelihood-ratio-based framework that measures the evidential strength of textual explanations. Our findings highlight fundamental limitations of current MLLMs for explainable face recognition and underscore the need for a principled evaluation of reliable and trustworthy explanations in biometric applications. Code is available at https://github.com/redwankarimsony/LR-MLLMFR-Explainability.

Figures (7)

• Figure 1: Example of GPT-4o openai2024gpt4o explanation for face verification. Although the model predicts the correct match decision, the highlighted text shows that the explanation relies on non-verifiable attributes, indicating weak visual grounding.
• Figure 2: Training and testing pipeline for LR-based evaluation of MLLM explanations. During training (top), explanations generated with ground-truth labels are embedded and used to learn class-conditional GMMs for genuine ($H_0$) and impostor ($H_1$) hypotheses; trainable modules are marked , and frozen modules are marked . During testing (bottom), explanations generated under different prompting regimes, with or without auxiliary FR information, are embedded using the frozen pipeline and evaluated via text-derived likelihood ratios.
• Figure 3: Row-normalized confusion matrices showing the impact of auxiliary FR information on MLLM verification decisions and uncertainty handling.
• Figure 4: Examples of genuine pairs predicted as uncertain by GPT-4o despite ground-truth supervision, illustrating failures under extreme pose variation.
• Figure 5: t-SNE visualizations of feature embeddings under different prompt strategies. The six FR model used here are ArcFace deng2019arcface, AdaFace kim2022adaface, MagFace meng2021magface, FaceNet-VGGFace schroff2015facenet, FaceNet-CasiaWebFace schroff2015facenet, KPRPE kprpe_2022_kim.