DeeCLIP: A Robust and Generalizable Transformer-Based Framework for Detecting AI-Generated Images

TL;DR

DeeCLIP tackles the brittleness of AI-generated image detectors when facing unseen generators and real-world degradations. It introduces a robust transformer-based framework that fuses high- and low-level features via DeeFuser, fine-tunes the CLIP-ViT backbone with LoRA, and uses triplet loss to sculpt a discriminative embedding space. Empirically, it achieves 89.00% mean accuracy across 19 test subsets spanning GAN and diffusion models, with strong robustness to blur and compression and 78.99% accuracy on a completely new dataset, surpassing prior methods while using fewer trainable parameters. The approach supports zero-shot adaptation and practical deployment due to its parameter-efficient design and open-source availability.

Abstract

This paper introduces DeeCLIP, a novel framework for detecting AI-generated images using CLIP-ViT and fusion learning. Despite significant advancements in generative models capable of creating highly photorealistic images, existing detection methods often struggle to generalize across different models and are highly sensitive to minor perturbations. To address these challenges, DeeCLIP incorporates DeeFuser, a fusion module that combines high-level and low-level features, improving robustness against degradations such as compression and blurring. Additionally, we apply triplet loss to refine the embedding space, enhancing the model's ability to distinguish between real and synthetic content. To further enable lightweight adaptation while preserving pre-trained knowledge, we adopt parameter-efficient fine-tuning using low-rank adaptation (LoRA) within the CLIP-ViT backbone. This approach supports effective zero-shot learning without sacrificing generalization. Trained exclusively on 4-class ProGAN data, DeeCLIP achieves an average accuracy of 89.00% on 19 test subsets composed of generative adversarial network (GAN) and diffusion models. Despite having fewer trainable parameters, DeeCLIP outperforms existing methods, demonstrating superior robustness against various generative models and real-world distortions. The code is publicly available at https://github.com/Mamadou-Keita/DeeCLIP for research purposes.

Figures (5)

• Figure 1: Fine-tuning approaches for AI-generated image detection. Compared to (a) full fine-tuning and (b) linear probing, our approach allows more effective and efficient adaption of CLIP-ViT for AI-generated image detection tasks.
• Figure 2: DeeCLIP's architecture. Given an image, deep and shallow features are extracted using CLIP-ViT (ViT-L/14) fine-tuned with LoRA. Then, the DeeFuser module fuses these features, capturing both semantic meaning and fine-grained textures. The fused representation is processed through a projection layer, where the output is fed into the classifier to distinguish real from fake images.
• Figure 3: (a) DeeFuser module to fuse both deep and shallow features from CLIP-ViT encoder. (b) Illustration of triplet loss: the anchor sample (blue) is randomly selected from either the positive (green) or negative (red) samples. If the anchor is from the positive class, it is optimized to be closer to other positive samples, further away from negative samples, and vice versa.
• Figure 4: t-SNE visualization of feature distributions for different generative models. The scatter plots illustrate the t-SNE embeddings of features extracted from real (green) and generated (red) images across various generative models, showing how well the features separate real from fake images.
• Figure 5: DeeCLIP's robustness under gaussian blur and JPEG compression, two common real-world degradations affecting AI-generated images.