Transfer Learning of Real Image Features with Soft Contrastive Loss for Fake Image Detection

TL;DR

This work addresses the challenge of detecting fake images forged by unknown models by shifting focus from model-specific artifacts to stable real-image characteristics called natural traces. It introduces Natural Trace Forensics (NTF), a two-stage framework that first learns homogeneous real-trace representations via self-supervised mapping and then applies soft-contrastive transfer learning to align real traces while separating fake images. The method leverages a diverse dataset comprising 6 GANs, 6 diffusion models, and multi-step forgeries to demonstrate strong generalization, achieving a mean average precision of $96.2\%$ and commercial-model accuracy above $78.4\%$, with robustness to common image transformations. This provides a practical, model-agnostic approach to fake image detection, highlighting the value of stable real-image traces over continually shifting artifact patterns.

Abstract

In the last few years, the artifact patterns in fake images synthesized by different generative models have been inconsistent, leading to the failure of previous research that relied on spotting subtle differences between real and fake. In our preliminary experiments, we find that the artifacts in fake images always change with the development of the generative model, while natural images exhibit stable statistical properties. In this paper, we employ natural traces shared only by real images as an additional target for a classifier. Specifically, we introduce a self-supervised feature mapping process for natural trace extraction and develop a transfer learning based on soft contrastive loss to bring them closer to real images and further away from fake ones. This motivates the detector to make decisions based on the proximity of images to the natural traces. To conduct a comprehensive experiment, we built a high-quality and diverse dataset that includes generative models comprising GANs and diffusion models, to evaluate the effectiveness in generalizing unknown forgery techniques and robustness in surviving different transformations. Experimental results show that our proposed method gives 96.2% mAP significantly outperforms the baselines. Extensive experiments conducted on popular commercial platforms reveal that our proposed method achieves an accuracy exceeding 78.4%, underscoring its practicality for real-world application deployment.

Figures (6)

• Figure 1: TSNE visualization.Left: The detector is easily overfitted to fake images in the training set; Right: Training with natural traces can generalize to unknown fake images.
• Figure 2: NTF architecture. We first decouple the feature representation of real images into homogeneous and heterogeneous features. Next, the homogeneous features $z^{hom}$ participate in SCL with the real and fake image features. Detector classifies real and fake images guided by the target of intra-class aggregation and inter-class separation. Better view in color.
• Figure 3: Ablation study on NTF architecture. All detectors were trained using the ProGAN and tested on other generative models. The designs of NTF architecture improve generalization ability. The red dotted line depicts chance performance.
• Figure 4: Ablation study on training data. All detectors trained on different data sources (ProGAN or LDM) or different numbers of classes of the ProGAN data source (20 classes, 12 classes, and 4 classes).
• Figure 5: Robustness to four image processing operations, i.e., Gaussian blur (a), JPEG compression (b), Gaussian Noise (c), Scaling (d).