Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLIP Embeddings for AI-Generated Image Detection: A Few-Shot Study with Lightweight Classifier

Ziyang Ou

TL;DR

This work investigates whether frozen CLIP visual embeddings inherently encode discriminative cues for AI-generated image detection and whether a lightweight, few-shot adaptation pipeline can generalize to new domains. By projecting images to a fixed $\mathbf z \in \mathbb{R}^{512}$ via a frozen CLIP encoder and training compact classifiers (MLP or 1D-CNN) on top, the method achieves about $95\%$ accuracy on the CIFAKE benchmark without end-to-end fine-tuning, and around $85\%$ accuracy in a 20% few-shot adaptation on a custom dataset. A zero-shot comparison using Google Gemini-2.0 highlights the limits of prompt-based detection for certain styles, though prompt engineering can recover some performance. The results reveal a tangible real-vs-fake boundary within CLIP’s latent space, demonstrate a computationally efficient detection pipeline, and highlight style- and texture-related challenges that guide future work toward broader domain coverage and high-resolution texture analysis.

Abstract

Verifying the authenticity of AI-generated images presents a growing challenge on social media platforms these days. While vision-language models (VLMs) like CLIP outdo in multimodal representation, their capacity for AI-generated image classification is underexplored due to the absence of such labels during the pre-training process. This work investigates whether CLIP embeddings inherently contain information indicative of AI generation. A proposed pipeline extracts visual embeddings using a frozen CLIP model, feeds its embeddings to lightweight networks, and fine-tunes only the final classifier. Experiments on the public CIFAKE benchmark show the performance reaches 95% accuracy without language reasoning. Few-shot adaptation to curated custom with 20% of the data results in performance to 85%. A closed-source baseline (Gemini-2.0) has the best zero-shot accuracy yet fails on specific styles. Notably, some specific image types, such as wide-angle photographs and oil paintings, pose significant challenges to classification. These results indicate previously unexplored difficulties in classifying certain types of AI-generated images, revealing new and more specific questions in this domain that are worth further investigation.

CLIP Embeddings for AI-Generated Image Detection: A Few-Shot Study with Lightweight Classifier

TL;DR

This work investigates whether frozen CLIP visual embeddings inherently encode discriminative cues for AI-generated image detection and whether a lightweight, few-shot adaptation pipeline can generalize to new domains. By projecting images to a fixed via a frozen CLIP encoder and training compact classifiers (MLP or 1D-CNN) on top, the method achieves about accuracy on the CIFAKE benchmark without end-to-end fine-tuning, and around accuracy in a 20% few-shot adaptation on a custom dataset. A zero-shot comparison using Google Gemini-2.0 highlights the limits of prompt-based detection for certain styles, though prompt engineering can recover some performance. The results reveal a tangible real-vs-fake boundary within CLIP’s latent space, demonstrate a computationally efficient detection pipeline, and highlight style- and texture-related challenges that guide future work toward broader domain coverage and high-resolution texture analysis.

Abstract

Verifying the authenticity of AI-generated images presents a growing challenge on social media platforms these days. While vision-language models (VLMs) like CLIP outdo in multimodal representation, their capacity for AI-generated image classification is underexplored due to the absence of such labels during the pre-training process. This work investigates whether CLIP embeddings inherently contain information indicative of AI generation. A proposed pipeline extracts visual embeddings using a frozen CLIP model, feeds its embeddings to lightweight networks, and fine-tunes only the final classifier. Experiments on the public CIFAKE benchmark show the performance reaches 95% accuracy without language reasoning. Few-shot adaptation to curated custom with 20% of the data results in performance to 85%. A closed-source baseline (Gemini-2.0) has the best zero-shot accuracy yet fails on specific styles. Notably, some specific image types, such as wide-angle photographs and oil paintings, pose significant challenges to classification. These results indicate previously unexplored difficulties in classifying certain types of AI-generated images, revealing new and more specific questions in this domain that are worth further investigation.
Paper Structure (19 sections, 1 equation, 7 figures)

This paper contains 19 sections, 1 equation, 7 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. CLIP Embedding + MLP / CNN on CIFAKE dataset
  5. Few‑Shot Adaptation: CLIP Embedding + MLP / CNN on custom dataset
  6. Few‑shot split.
  7. Training details.
  8. Why few‑shot is feasible.
  9. Closed-source prompt-based multimodal LLM test
  10. Prompt design.
  11. Roles in this study.
  12. Experiments
  13. Dataset
  14. Evaluation pipeline CLIP Embedding + MLP / CNN on CIFAKE dataset
  15. Few-shot on custom dataset
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Sample images from the two datasets.
  • Figure 2: Performance metrics based on CIFAKE dataset. Achieves $\approx95\%$ Accuracy, Precision, and Recall on a large test set (12 000 samples), far exceeding random guessing.
  • Figure 3: Performance metrics based on custom dataset. MLP: light classifier, indicating a "linearly separable" baseline. ConvNet: captures different hidden patterns in the embedding dimension.
  • Figure 4: Misclassifications examples. Commons are shared among misclassified images in the custom dataset: wide-view landmarks and oil paintings.
  • Figure 5: Google Gemini's response on a single fake image, detailed response but wrong prediction
  • ...and 2 more figures