Detecting AI-Generated Images via Distributional Deviations from Real Images

TL;DR

This work analyzes frozen CLIP-ViT representations for AI-generated image detection and finds real images form a compact high-level cluster while AI-generated images remain outliers. It introduces Masking-based Pre-trained model Fine-Tuning (MPFT) with a Texture-Aware Masking (TAM) mechanism to mask texture-rich regions during fine-tuning, forcing the model to rely on distributional deviations in the high-level feature space. By fine-tuning CLIP-ViT with only a small amount of data, MPFT achieves superior generalization to unseen generative models, outperforming existing methods on GenImage and UniversalFakeDetect (up to 98.2% Avg. Acc and 94.6% Avg. Acc, respectively). The approach demonstrates robust cross-model performance, with CAM analyses showing refined attention to real-image regions and reduced focus on model-specific artifacts, suggesting a practical, scalable path for trustworthy AI-content verification.

Abstract

The rapid advancement of generative models has significantly enhanced the quality of AI-generated images, raising concerns about misinformation and the erosion of public trust. Detecting AI-generated images has thus become a critical challenge, particularly in terms of generalizing to unseen generative models. Existing methods using frozen pre-trained CLIP models show promise in generalization but treat the image encoder as a basic feature extractor, failing to fully exploit its potential. In this paper, we perform an in-depth analysis of the frozen CLIP image encoder (CLIP-ViT), revealing that it effectively clusters real images in a high-level, abstract feature space. However, it does not truly possess the ability to distinguish between real and AI-generated images. Based on this analysis, we propose a Masking-based Pre-trained model Fine-Tuning (MPFT) strategy, which introduces a Texture-Aware Masking (TAM) mechanism to mask textured areas containing generative model-specific patterns during fine-tuning. This approach compels CLIP-ViT to attend to the "distributional deviations"from authentic images for AI-generated image detection, thereby achieving enhanced generalization performance. Extensive experiments on the GenImage and UniversalFakeDetect datasets demonstrate that our method, fine-tuned with only a minimal number of images, significantly outperforms existing approaches, achieving up to 98.2% and 94.6% average accuracy on the two datasets, respectively.

Figures (7)

• Figure 1: t-SNE van2008visualizing visualization of features extracted by pre-trained CLIP-ViT from images generated by eight models in the GenImage dataset. (a) Frozen CLIP-ViT. (b) CLIP-ViT fine-tuned with only SDV1.4-generated images. (c) CLIP-ViT fine-tuned with images generated by all involved generative models. (d) CLIP-ViT fine-tuned with our proposed MPFT using only SDV1.4-generated images.
• Figure 2: Left: Generalization performance (average accuracy across all 8 subsets of the GenImage test set) of three models: frozen CLIP-ViT with a trainable linear classifier, directly fine-tuned CLIP-ViT, and CLIP-ViT fine-tuned using our MPFT strategy. All models are trained on the SDV1.4 subset of the GenImage training set. Right: Accuracy of the models on the SDV1.4 (seen), ADM (unseen), and BigGAN (unseen) subsets of the GenImage test set.
• Figure 3: Overview of the Fine-tuning and Testing Pipeline.
• Figure 4: Accuracy (Acc) Across 8 Subsets on GenImage: Comparison of CLIP-ViT with two fine-tuning strategies: direct fine-tuning (base) and fine-tuning with our proposed MPFT method. Models are fine-tuned on a single subset and evaluated on all 8 subsets. The color scale represents performance, with darker shades indicating higher accuracy.
• Figure 5: Average Acc and average AP of CLIP-ViT fine-tuned with our method using different numbers of images (400, 800, 3200, 12800, and 25600)—with an equal split between real and generated images—evaluated on the GenImage and UniversalFakeDetect. To reflect the stability of the model’s generalization, we report the results averaged over the last five epochs.