2D-3D Attention and Entropy for Pose Robust 2D Facial Recognition

TL;DR

This paper tackles pose variability in 2D facial recognition by training 2D embeddings under the guidance of pose-invariant 3D priors. It introduces a 2D-3D Joint Attention Mapping (JAM) with a shared attention mechanism and a Joint Entropy (JE) regularizer to align 2D and 3D representations during training, while enabling purely 2D inference. The approach leverages asymmetric backbones (2D IR-Net-18 and 3D PointNet++) and AdaFace-based identification, optimized with a joint loss L = L_{2D} + L_{3D} + L_{JE} and equations for attention and probabilities. Empirical evaluations on FaceScape and ARL-VTF show improved pose-robust performance, with notable TAR@1%FAR gains for extreme poses and good generalization across datasets, highlighting the method's practical impact for unconstrained facial recognition. The work emphasizes the benefit of learning 3D priors offline to enhance 2D perception without adding 3D data during deployment, and discusses ethical considerations and dataset constraints for real-world use.

Abstract

Despite recent advances in facial recognition, there remains a fundamental issue concerning degradations in performance due to substantial perspective (pose) differences between enrollment and query (probe) imagery. Therefore, we propose a novel domain adaptive framework to facilitate improved performances across large discrepancies in pose by enabling image-based (2D) representations to infer properties of inherently pose invariant point cloud (3D) representations. Specifically, our proposed framework achieves better pose invariance by using (1) a shared (joint) attention mapping to emphasize common patterns that are most correlated between 2D facial images and 3D facial data and (2) a joint entropy regularizing loss to promote better consistency$\unicode{x2014}$enhancing correlations among the intersecting 2D and 3D representations$\unicode{x2014}$by leveraging both attention maps. This framework is evaluated on FaceScape and ARL-VTF datasets, where it outperforms competitive methods by achieving profile (90$\unicode{x00b0}$$\unicode{x002b}$) TAR @ 1$\unicode{x0025}$ FAR improvements of at least 7.1$\unicode{x0025}$ and 1.57$\unicode{x0025}$, respectively.

Figures (5)

• Figure 1: Conventional 2D image embedding networks for FR often exhibit mismatches due to not being invariant to pose. Instead, we introduce a framework that implicitly infers pose invariant 3D information when extracting 2D image embeddings via our proposed 2D-3D Joint Attention Mapping (JAM) and 2D-3D Joint Entropy regularizing loss. This framework processes 2D images (left) to output image representations (right) that are inherently robust to pose variations. For example, images A and B yield similar representations from the same identity, while B and D yield distinct ones despite a similar pose, as they depict different identities. Subjects consented for publication.
• Figure 2: Our FR framework models pose-invariance by introducing JAM and JE, where this combination promotes consistency of 2D and 3D representations indirectly by minimizing the joint entropy over the attention maps ($A_{2D}$ and $A_{3D}$). While both 2D and 3D data are used during training (Red and Black arrows), only 2D images are utilized during inference. Subjects consented for publication.
• Figure 3: ROC Curves for frontal and profile poses: (a) Frontal pose evaluation of embedding networks with AUC and EER. (b) Profile pose analysis using the same metrics. Shaded regions show variability, representing one standard deviation from the mean.
• Figure 4: t-SNE plots comparing (a) CrossPoint (a) and Ours + JAM + JE (b). In the legend (c) subjects IDs are indicated by color and poses are indicated by shape (circles for frontal, squares for profile). Ours + JAM + JE shows better separability across identity and pose. Notably, IDs: [0, 4, 5, 12, 19] mark improvement with closer convergence of pose embeddings, demonstrating consistency of our method across poses.
• Figure 5: Comparison of attention maps superimposed on facial images for frontal and profile pairs across three methods. Green outlines indicate matches above the TAR at 1% FAR threshold, while Red shows non-matches. The color bar ranges from red (high attention) to blue (low attention). The arrow shows pose difference from $60^{\circ}$ to $90^{\circ}$ (top to bottom). Ours + JAM + JE exhibits more consistent attention maps across poses and better regularizes attention regions. However, Ours + JAM enhances regions of focus, while Ours + JAM + JE de-emphasizes areas prone to misclassification. Subjects consented for publication.