Model Discrepancy Learning: Synthetic Faces Detection Based on Multi-Reconstruction

TL;DR

The paper tackles the challenge of detecting synthetic faces generated by GANs and diffusion models by analyzing reconstruction discrepancies across generation techniques. It introduces a Multi-Reconstruction-based Detector that inverts and reconstructs inputs with both GAN and diffusion models, then classifies using cascaded original and reconstructed images, achieving strong generalization to unseen generators. To support this line of work, the authors present the Asian Synthetic Face Dataset (ASFD), addressing underrepresentation of Asian populations in existing datasets. Empirical results demonstrate superior accuracy and robustness compared with existing detectors, including across cross-dataset scenarios and perturbations, highlighting the practical value of model-discrepancy-aware detection for real-world security tasks.

Abstract

Advances in image generation enable hyper-realistic synthetic faces but also pose risks, thus making synthetic face detection crucial. Previous research focuses on the general differences between generated images and real images, often overlooking the discrepancies among various generative techniques. In this paper, we explore the intrinsic relationship between synthetic images and their corresponding generation technologies. We find that specific images exhibit significant reconstruction discrepancies across different generative methods and that matching generation techniques provide more accurate reconstructions. Based on this insight, we propose a Multi-Reconstruction-based detector. By reversing and reconstructing images using multiple generative models, we analyze the reconstruction differences among real, GAN-generated, and DM-generated images to facilitate effective differentiation. Additionally, we introduce the Asian Synthetic Face Dataset (ASFD), containing synthetic Asian faces generated with various GANs and DMs. This dataset complements existing synthetic face datasets. Experimental results demonstrate that our detector achieves exceptional performance, with strong generalization and robustness.

Figures (6)

• Figure 1: GANs and DMs achieve higher reconstruction quality for data aligned with their distributions, while the quality degrades for data outside these distributions. Specifically, well-reconstructed images are near the center of the distribution, whereas poorly reconstructed ones lie at the periphery.
• Figure 2: Reconstructed images of different categories of images. The first row shows the original images, the second row displays the images reconstructed by GAN, and the third row presents the images reconstructed by DM.
• Figure 3: Multi-Reconstruction-based Detector. Given an input image $X$, the encoder extracts hierarchical latent codes $w$ and feature codes $w^{*}$, which are concatenated to form the latent representation $Z_{R}$ in the GAN's latent space. This representation is fed into a pre-trained StyleGAN to obtain the reconstructed image $X_{RG}$. Simultaneously, the input image is inverted to the latent representation $Z_{T}$ in a DM's latent space using DDIM inversion. The reconstructed image $X_{RD}$ is then obtained through the DDIM denoising process. Finally, $X$, $X_{RG}$ and $X_{RD}$ are input into a ResNet-50 network for further processing.
• Figure 4: Asian Synthetic Face Dataset. The first row contains images generated using different GANs, while the second row contains images generated using different DMs.
• Figure 5: Robustness evaluations against Gaussian blur and JPEG compression.