When Semantics Regulate: Rethinking Patch Shuffle and Internal Bias for Generated Image Detection with CLIP

TL;DR

The paper tackles AI-generated image detection under distribution shifts and shows that semantic bias undermines cross-domain robustness. It reveals that Patch Shuffle disrupts global semantics while preserving local artifact cues, making CLIP's semantic manifold a regulator of representations and enabling artifact-focused learning. The authors propose SemAnti, which freezes CLIP's semantic subspace and fine-tunes only artifact-sensitive layers under shuffled semantics, achieving state-of-the-art cross-domain performance on GenImage and AIGCDetectBenchmark. The results demonstrate that controlling semantics is key to unlocking CLIP's robustness for forgery detection, with insights supported by layer-wise analyses and visualizations. This approach promises more reliable AI-generated image detection across diverse generators and domains.

Abstract

The rapid progress of GANs and Diffusion Models poses new challenges for detecting AI-generated images. Although CLIP-based detectors exhibit promising generalization, they often rely on semantic cues rather than generator artifacts, leading to brittle performance under distribution shifts. In this work, we revisit the nature of semantic bias and uncover that Patch Shuffle provides an unusually strong benefit for CLIP, that disrupts global semantic continuity while preserving local artifact cues, which reduces semantic entropy and homogenizes feature distributions between natural and synthetic images. Through a detailed layer-wise analysis, we further show that CLIP's deep semantic structure functions as a regulator that stabilizes cross-domain representations once semantic bias is suppressed. Guided by these findings, we propose SemAnti, a semantic-antagonistic fine-tuning paradigm that freezes the semantic subspace and adapts only artifact-sensitive layers under shuffled semantics. Despite its simplicity, SemAnti achieves state-of-the-art cross-domain generalization on AIGCDetectBenchmark and GenImage, demonstrating that regulating semantics is key to unlocking CLIP's full potential for robust AI-generated image detection.

Figures (7)

• Figure 1: Patch Shuffle (PS) alleviates the semantic imbalance between natural and synthetic images. Before PS (left), natural samples exhibit richer and more diverse semantics, forming a wide and dispersed cluster, whereas synthetic samples remain concentrated, leading to a biased separation. After PS (right), PS homogenizes the distributions of both classes, enabling a more balanced and artifact-oriented decision boundary.
• Figure 2: Accuracy of five detectors under different Patch Shuffle (PS) patch sizes (PS=0 denotes no shuffle). Only CLIP shows significant improvement, indicating that PS effectively mitigates its reliance on semantics and enhances artifact-level generalization.
• Figure 3: Overview of our architecture. Patch Shuffle homogenizes image semantics, balancing the representation distributions of natural and synthetic samples and steering the model toward local artifact cues. In addition, we freeze the semantic-rich deep layers of CLIP to preserve high-level semantic artifact priors, enabling more robust cross-domain generalization.
• Figure 4: Layer-wise fine-tuning analysis of CLIP. Fine-tuning shallow layers enhances artifact sensitivity, while deeper layers lead to overfitting to semantic cues.
• Figure 5: Visualization of the latent distributions of the original CLIP and our method under different settings. More visualizations are provided in the appendix.