Artifact Feature Purification for Cross-domain Detection of AI-generated Images

TL;DR

The proposed Artifact Purification Network (APN) to facilitate the artifact extraction from generated images through the explicit and implicit purification processes finds that the proposed method extracts flexible forgery patterns and condenses the forgery information diluted in irrelevant features.

Abstract

In the era of AIGC, the fast development of visual content generation technologies, such as diffusion models, bring potential security risks to our society. Existing generated image detection methods suffer from performance drop when faced with out-of-domain generators and image scenes. To relieve this problem, we propose Artifact Purification Network (APN) to facilitate the artifact extraction from generated images through the explicit and implicit purification processes. For the explicit one, a suspicious frequency-band proposal method and a spatial feature decomposition method are proposed to extract artifact-related features. For the implicit one, a training strategy based on mutual information estimation is proposed to further purify the artifact-related features. Experiments show that for cross-generator detection, the average accuracy of APN is 5.6% ~ 16.4% higher than the previous 10 methods on GenImage dataset and 1.7% ~ 50.1% on DiffusionForensics dataset. For cross-scene detection, APN maintains its high performance. Via visualization analysis, we find that the proposed method extracts flexible forgery patterns and condenses the forgery information diluted in irrelevant features. We also find that the artifact features APN focuses on across generators and scenes are global and diverse. The code will be available on GitHub.

Figures (6)

• Figure 1: The cross-domain generated image detection. (a) The performance of CNNSpot wang2020cnn, F3Net qian2020thinking, GramNet liu2020global, Swin-T liu2021swin, Patch5M ju2022fusing, SIA sun2022information, LNP liu2022detecting, LGrad tan2023learning and ours (APN). They are trained on SD v1-generated samples under ImageNet scene deng2009imagenet. Then they are tested on GLIDE-generated samples under ImageNet scene (cross-generator setting), and on SD v1-generated samples under LSUN-Bedroom scene yu2015lsun (cross-scene setting). (b) The distribution difference between images of different domains, and the comparison between existing methods and ours.
• Figure 2: The framework of Artifact Purification Network (APN). APN facilitates the artifact extraction process through the explicit and the implicit purification. The explicit one separates artifacts and extracts the features from frequency and spatial modalities, where a suspicious frequency-band proposal method and a learnable spatial feature decomposition method are proposed for them respectively. The implicit one further purifies the artifact-related features based on the mutual information estimation.
• Figure 3: The visualization of artifact-related and -irrelated features of samples in different scene and generator domains with / without the implicit purification using t-SNE. The top row: APN with the implicit purification. The bottom row: APN without the implicit purification. Please zoom in to see the details.
• Figure 4: The differences in proposals across generators and scenes. (a) and (c): the confidence-weighted and normalized frequencies of the proposals. (b) and (d): the correlation matrices of the curves in (a) and (c). (a) and (b) correspond to the fake images in GenImage subsets. (c) and (d) correspond to the fake images generated by SD v1.4 in ImageNet (GenImage) and LSUN-Bedroom (DF) scene.
• Figure 5: The images and their spatial heatmaps of GenImage (ImageNet scene) and DF (LSUN-Bedroom scene) samples. The predicted probabilities are below the images. The images in each column in ImageNet scene have the same class label.