Exploiting Multiple Representations: 3D Face Biometrics Fusion with Application to Surveillance

TL;DR

This work addresses robust face verification in surveillance by exploiting complementary information from multiple 3D face reconstruction (3DFR) algorithms. It trains several recognition systems on diverse 3DFR-derived representations and fuses their scores using non-parametric, weighted, and classification-based fusion strategies to improve robustness across intra-, cross-, and cross-dataset settings. Experiments on SCFace and cross-dataset data show that fusion can significantly boost performance and stability, particularly under challenging acquisition variations, and provide actionable guidelines for practitioners. The findings demonstrate the practicality of ensemble 3DFR representations for real-world surveillance tasks and offer avenues for extending fusion-backed 3DFR methods to broader facial biometrics applications.

Abstract

3D face reconstruction (3DFR) algorithms are based on specific assumptions tailored to the limits and characteristics of the different application scenarios. In this study, we investigate how multiple state-of-the-art 3DFR algorithms can be used to generate a better representation of subjects, with the final goal of improving the performance of face recognition systems in challenging uncontrolled scenarios. We also explore how different parametric and non-parametric score-level fusion methods can exploit the unique strengths of multiple 3DFR algorithms to enhance biometric recognition robustness. With this goal, we propose a comprehensive analysis of several face recognition systems across diverse conditions, such as varying distances and camera setups, intra-dataset and cross-dataset, to assess the robustness of the proposed ensemble method. The results demonstrate that the distinct information provided by different 3DFR algorithms can alleviate the problem of generalizing over multiple application scenarios. In addition, the present study highlights the potential of advanced fusion strategies to enhance the reliability of 3DFR-based face recognition systems, providing the research community with key insights to exploit them in real-world applications effectively. Although the experiments are carried out in a specific face verification setup, our proposed fusion-based 3DFR methods may be applied to other tasks around face biometrics that are not strictly related to identity recognition.

Figures (11)

• Figure 1: Advantages of 3D face reconstruction from a 2D acquisition in comparison with 3D data acquisition.
• Figure 2: Proposed method. After the 3DFR algorithms, the synthetic view generation module creates 2D images ($\textbf{x}_i$) from the 3D templates derived from the 2D reference image. These synthetic views represent different perspectives of the reference face to enhance recognition. The 2D probe image ($\textbf{y}$) is matched against each representation to determine similarity. In addition to our proposed 3DFR fusion scheme, we also depict the main elements of our experimental setup based on the SCface dataset together with example high-quality reference and low-quality probe images at the three distances considered (probe images after face detection). For more details of the experimental setup see the original dataset description grgic2011scface and related works tome2013tome2015.
• Figure 3: Examples of personalized 3D templates generated from a mugshot (a) in the SCface dataset grgic2011scface using EOS eos (b), NextFace dib2021practical (c), 3DDFA V2 guo2020towards, and HRN lei2023hierarchical (e).
• Figure 4: Taxonomy of the analyzed score-level fusion methods.
• Figure 5: Siamese Neural Network module overview: the 3D reference image and 2D probe image undergo face detection followed by feature extraction through a shared backbone network. Note that, during testing, we only generate frontal faces from the reference image, which are then compared with the corresponding set of probe images. The embeddings are compared using a mated comparison score measure to compute the likelihood $P(\text{mated}|\textbf{x}_i,\textbf{y})$, indicating whether the two inputs correspond to the same individual.