Synthetic to Authentic: Transferring Realism to 3D Face Renderings for Boosting Face Recognition

TL;DR

This work tackles privacy and data-collection constraints in face recognition by leveraging synthetic 3D-rendered faces, which historically underperform compared with real-data-trained models. It proposes a realism-transfer framework that applies image-to-image translation and face restoration to produce photorealistic renders without identity labels or pre-trained FR guidance. Through multiple realism-transfer methods, the authors demonstrate FR performance gains of roughly 2–5% on standard benchmarks, reducing the gap to real-data baselines. The findings indicate that synthetic data, when rendered with photorealism, can effectively train FR systems for real-world deployment, offering privacy-friendly and scalable data-generation options.

Abstract

In this paper, we investigate the potential of image-to-image translation (I2I) techniques for transferring realism to 3D-rendered facial images in the context of Face Recognition (FR) systems. The primary motivation for using 3D-rendered facial images lies in their ability to circumvent the challenges associated with collecting large real face datasets for training FR systems. These images are generated entirely by 3D rendering engines, facilitating the generation of synthetic identities. However, it has been observed that FR systems trained on such synthetic datasets underperform when compared to those trained on real datasets, on various FR benchmarks. In this work, we demonstrate that by transferring the realism to 3D-rendered images (i.e., making the 3D-rendered images look more real), we can boost the performance of FR systems trained on these more photorealistic images. This improvement is evident when these systems are evaluated against FR benchmarks utilizing real-world data, thereby paving new pathways for employing synthetic data in real-world applications.

Figures (4)

• Figure 1: 3D-Rendered images of human faces bae2023digiface (left image in each column), and post-processing images by image-to-image translation (right image in each column) for boosting the performance of a Face Recognition trained on the synthetic data.
• Figure 2: In this paper, we study the efficacy of image-to-image translation methodologies applied to enhance the performance of face recognition---in essence a challenging classification task. Starting with a dataset of 3D-rendered human faces (i.e.$\mathcal{D}_{\mathsf{syn}}$), that exhibit a domain shift compared to real-world human face images, we apply various image-to-image translation and Face Restoration methodologies (i.e., Realism Transfer Method Block) that only require limited identity unlabeled real datasets (i.e., $\mathcal{D}_{\mathsf{real}}$) or subset of unrealistic images (i.e., $\mathcal{D}_{\mathsf{syn}}^{\prime}$) themselves to train. We then train a face recognition network on both the original (unrealistic-looking) and the newly translated (more realistic) images, $\mathcal{D}_{\mathsf{RT}}$, to investigate whether this approach can improve the accuracy of FR systems.
• Figure 3: From the left to right, the first column corresponds to the original DigiFace1M dataset, and the next columns are from after applying different translation tasks to the original images, CodeFormer, VSAIT, DECENT, UNSB-NE-1, UNSB-NE-5 and DDIM Inversion, respectively.
• Figure 4: ROC Curve on the IJB-C benchmark, for each dataset we selected one of the models in which we trained an FR on top of it, and plotted the ROC curve.