Multimodal Conditional Information Bottleneck for Generalizable AI-Generated Image Detection

TL;DR

This work tackles the generalization gap in CLIP-based AI-generated image detectors caused by feature redundancy. It introduces InfoFD, a multimodal conditional information bottleneck framework that conditions latent representations on text and class information, using Text-Guided Conditional Information Bottleneck (TGCIB) and Dynamic Text Orthogonalization (DTO) to reduce redundancy and exploit text signals. The method employs a variational approximation with Gram-Schmidt orthogonalization and a training objective combining a maximum-mean-discrepancy term with a classification loss, alongside Composite Gaussian Perturbation to simulate multi-domain variations. Empirical results on GenImage and CO-SPY demonstrate strong cross-model generalization without CLIP fine-tuning, with ablations confirming the benefit of CGP and the text-conditioned approach. The work highlights a textual bias in CLIP features that can be harnessed for improved detection and provides a robust pipeline for generalized AI-generated image detection.

Abstract

Although existing CLIP-based methods for detecting AI-generated images have achieved promising results, they are still limited by severe feature redundancy, which hinders their generalization ability. To address this issue, incorporating an information bottleneck network into the task presents a straightforward solution. However, relying solely on image-corresponding prompts results in suboptimal performance due to the inherent diversity of prompts. In this paper, we propose a multimodal conditional bottleneck network to reduce feature redundancy while enhancing the discriminative power of features extracted by CLIP, thereby improving the model's generalization ability. We begin with a semantic analysis experiment, where we observe that arbitrary text features exhibit lower cosine similarity with real image features than with fake image features in the CLIP feature space, a phenomenon we refer to as "bias". Therefore, we introduce InfoFD, a text-guided AI-generated image detection framework. InfoFD consists of two key components: the Text-Guided Conditional Information Bottleneck (TGCIB) and Dynamic Text Orthogonalization (DTO). TGCIB improves the generalizability of learned representations by conditioning on both text and class modalities. DTO dynamically updates weighted text features, preserving semantic information while leveraging the global "bias". Our model achieves exceptional generalization performance on the GenImage dataset and latest generative models. Our code is available at https://github.com/Ant0ny44/InfoFD.

Figures (16)

• Figure 1: Illustration of Multimodal Conditional Information Bottleneck and T-SNE visualization of "bias".(a)(I) The standard Information Bottleneck (IB) vib2017. (II) The Conditional Information Bottleneck (CIB) cib2020. (III) Our MultiModal Conditional Information Bottleneck (MMCIB). (b) Blue triangles: fake images. Green circles: real images. Red triangles: prompts for fake images. Yellow circles: prompts for real images. Image features are obtained from the [CLS] output of CLIP's layer11 followed by projection through the final layer, while text features are encoded using CLIP's text encoder. The real images and their corresponding text are sourced from ImageNet imagenet2009, while the fake images and their corresponding text are obtained from Stable Diffusion diffusiondb2022. The centroids corresponding to each category are marked with pentagrams.
• Figure 2: (a)Mean $cos (\cdot, \cdot)$ of different datasets with the same pooling random text features $\tilde{t}$. Here, $I_r$ denotes real image features encoded by layer11, $I_f$ denotes fake image features, and $\tilde{t}$ consists of 1,000 randomly generated strings with lengths between 30 and 70, composed of randomly mixed uppercase and lowercase letters, using $3-\sigma$ error bars. All images are sourced from ImageNet imagenet2009. For more detailed experimental results, please refer to \ref{['sec:appendix_exp_preliminaries']}. (b)Outputs of different blocks in CLIP's ViT encoder. Real images from ImageNet and fake images from Stable Diffusion V1.5 in GenImage, with 2000 images each.
• Figure 3: Pipeline of infoFD. InfoFD is composed of two components: prompts processing and image processing. During the text processing phase, the texts are first categorized into real and fake groups based on their corresponding image classes. DTO: Subsequently, we employ DTO for information integration, ultimately obtaining $r(z \mid t, y)$. We directly utilize the first 12 layers of the ViT in CLIP as the image encoder, based on our findings in the Preliminaries section. TGCIB: After being encoded by CLIP's ViT, the image features are augmented using Composite Gaussian Perturbation (CGP) and then used reparameterization to simulate sampling and $r(z \mid t, y)$ to guide optimization. Here, $\epsilon_g$ is composite noise generated by combining Gaussian and uniform distributions.
• Figure 4: (a)$I_D(X;Z)$ against $I_D(Z;Y)$ on MMCIB and IB. The black line represents the trend of MMCIB obtained through triple curve fitting. The results are calculated on GenImage genimage2024 Test set. (b) Illustration of feature redundancy. The darker the color, the greater the degree of compression. The baseline is controlled using different sizes of hidden dimensions.
• Figure 5: PCA Projection.Green: real image features; Red: fake image features; CLIP: CLIP feature projections; $z$: TGIB-extracted representations.