Beyond Real Faces: Synthetic Datasets Can Achieve Reliable Recognition Performance without Privacy Compromise

TL;DR

This work systematically evaluates synthetic facial recognition data as a privacy-preserving alternative to real datasets. It combines a literature review of 25 datasets with extensive experiments across millions of synthetic samples, framed by seven privacy-oriented requirements. The analysis identifies top-performing synthetic datasets (VariFace, VIGFace) that approach or exceed real-dataset benchmarks, while publicly accessible options (Vec2Face, CemiFace) closely follow. The study also highlights gaps in bias mitigation, the need for standardized synthetic benchmarks, and pathways toward fully synthetic, consent-free training and evaluation pipelines with robust ethical safeguards.

Abstract

The deployment of facial recognition systems has created an ethical dilemma: achieving high accuracy requires massive datasets of real faces collected without consent, leading to dataset retractions and potential legal liabilities under regulations like GDPR. While synthetic facial data presents a promising privacy-preserving alternative, the field lacks comprehensive empirical evidence of its viability. This study addresses this critical gap through extensive evaluation of synthetic facial recognition datasets. We present a systematic literature review identifying 25 synthetic facial recognition datasets (2018-2025), combined with rigorous experimental validation. Our methodology examines seven key requirements for privacy-preserving synthetic data: identity leakage prevention, intra-class variability, identity separability, dataset scale, ethical data sourcing, bias mitigation, and benchmark reliability. Through experiments involving over 10 million synthetic samples, extended by a comparison of results reported on five standard benchmarks, we provide the first comprehensive empirical assessment of synthetic data's capability to replace real datasets. Best-performing synthetic datasets (VariFace, VIGFace) achieve recognition accuracies of 95.67% and 94.91% respectively, surpassing established real datasets including CASIA-WebFace (94.70%). While those images remain private, publicly available alternatives Vec2Face (93.52%) and CemiFace (93.22%) come close behind. Our findings reveal that they ensure proper intra-class variability while maintaining identity separability. Demographic bias analysis shows that, even though synthetic data inherits limited biases, it offers unprecedented control for bias mitigation through generation parameters. These results establish synthetic facial data as a scientifically viable and ethically imperative alternative for facial recognition research.

Figures (9)

• Figure 1: Publications overview
• Figure 2: Timeline of first public appearance of the identified synthetic datasets. Datasets released in the same month are aligned vertically.
• Figure 3: Dataset generation method by year
• Figure 4: Exemplary images of an identity across synthetic datasets
• Figure 5: Cosine similarities distribution of closest samples with respect to CASIA-WebFace dataset.