Generalizable Synthetic Image Detection via Language-guided Contrastive Learning

TL;DR

This work tackles the weak cross-domain generalization of synthetic image detectors by introducing LASTED, a language-guided contrastive learning framework that augments training images with carefully designed textual labels to form a joint visual-language feature space. After training with a multimodal objective, a lightweight linear probe is used for detection, yielding strong generalization to unseen generation models across painting and photo styles. Extensive experiments across GANs, diffusion models, fused datasets, and real-world web data show LASTED outperforms state-of-the-art detectors and exhibits robustness to post-processing. The results underscore the value of multimodal supervision for forensic robustness with practical implications for online moderation, digital forensics, and combating misinformation.

Abstract

The heightened realism of AI-generated images can be attributed to the rapid development of synthetic models, including generative adversarial networks (GANs) and diffusion models (DMs). The malevolent use of synthetic images, such as the dissemination of fake news or the creation of fake profiles, however, raises significant concerns regarding the authenticity of images. Though many forensic algorithms have been developed for detecting synthetic images, their performance, especially the generalization capability, is still far from being adequate to cope with the increasing number of synthetic models. In this work, we propose a simple yet very effective synthetic image detection method via a language-guided contrastive learning. Specifically, we augment the training images with carefully-designed textual labels, enabling us to use a joint visual-language contrastive supervision for learning a forensic feature space with better generalization. It is shown that our proposed LanguAge-guided SynThEsis Detection (LASTED) model achieves much improved generalizability to unseen image generation models and delivers promising performance that far exceeds state-of-the-art competitors over four datasets. The code is available at https://github.com/HighwayWu/LASTED.

Figures (10)

• Figure 1:
• Figure 2: Illustration of our proposed LASTED. The training images are first augmented with the carefully-designed textual labels, and then image/text encoders are jointly trained.
• Figure 3: Existing detectors (e.g., Wang easyspot2020, HiFi hifi, UniDet unidet) suffer from severe performance degradation in out-of-domain scenarios (Midjourney midjourney, DreamBooth dreambooth, SAN san).
• Figure 4: Different training paradigms (MoCo moco, MAE mae, and CLIP clip) lead to different generalizability. It is worth noting that all models use the same ResNet50 resnet2016 architecture and are trained on the natural dataset ImageNet deng2009imagenet without fine-tuning. The testing is carried out on unseen real painting images (Danbooru danbooru2021) and the ones synthesized by Latent Diffusion latent_diffusion.
• Figure 5: Our proposed LASTED framework introduces a language-guided contrastive paradigm in its training process to better decouple the semantic information of the image and make the detector focus on the forensic signal. After the training is completed, the specific synthetic/real detection can be achieved by fine-tuning a lightweight linear probe.