Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Artificial Intelligence Generated Image Detection a Solved Problem?

Ziqiang Li, Jiazhen Yan, Ziwen He, Kai Zeng, Weiwei Jiang, Lizhi Xiong, Zhangjie Fu

TL;DR

Is Artificial Intelligence Generated Image detection a Solved Problem? introduces AIGIBench, a comprehensive benchmark to rigorously evaluate state-of-the-art AIGI detectors under realistic, real-world conditions. By assessing generalization to unseen generators, robustness to degradations, augmentation sensitivity, and test-time preprocessing across 23 diverse fake-image subsets plus real-world samples, the study finds that no detector consistently dominates and that performance degrades notably in real-world scenarios. The results highlight the need for robust, generalizable features—such as frequency-domain cues and CLIP-based representations—and more diverse training data to improve detection reliability in practice. Overall, AIGIBench provides a standardized, multifaceted framework to drive future research toward dependable AIGI detection and informs policy and deployment considerations in the context of synthetic media risks.

Abstract

The rapid advancement of generative models, such as GANs and Diffusion models, has enabled the creation of highly realistic synthetic images, raising serious concerns about misinformation, deepfakes, and copyright infringement. Although numerous Artificial Intelligence Generated Image (AIGI) detectors have been proposed, often reporting high accuracy, their effectiveness in real-world scenarios remains questionable. To bridge this gap, we introduce AIGIBench, a comprehensive benchmark designed to rigorously evaluate the robustness and generalization capabilities of state-of-the-art AIGI detectors. AIGIBench simulates real-world challenges through four core tasks: multi-source generalization, robustness to image degradation, sensitivity to data augmentation, and impact of test-time pre-processing. It includes 23 diverse fake image subsets that span both advanced and widely adopted image generation techniques, along with real-world samples collected from social media and AI art platforms. Extensive experiments on 11 advanced detectors demonstrate that, despite their high reported accuracy in controlled settings, these detectors suffer significant performance drops on real-world data, limited benefits from common augmentations, and nuanced effects of pre-processing, highlighting the need for more robust detection strategies. By providing a unified and realistic evaluation framework, AIGIBench offers valuable insights to guide future research toward dependable and generalizable AIGI detection.Data and code are publicly available at: https://github.com/HorizonTEL/AIGIBench.

Is Artificial Intelligence Generated Image Detection a Solved Problem?

TL;DR

Is Artificial Intelligence Generated Image detection a Solved Problem? introduces AIGIBench, a comprehensive benchmark to rigorously evaluate state-of-the-art AIGI detectors under realistic, real-world conditions. By assessing generalization to unseen generators, robustness to degradations, augmentation sensitivity, and test-time preprocessing across 23 diverse fake-image subsets plus real-world samples, the study finds that no detector consistently dominates and that performance degrades notably in real-world scenarios. The results highlight the need for robust, generalizable features—such as frequency-domain cues and CLIP-based representations—and more diverse training data to improve detection reliability in practice. Overall, AIGIBench provides a standardized, multifaceted framework to drive future research toward dependable AIGI detection and informs policy and deployment considerations in the context of synthetic media risks.

Abstract

The rapid advancement of generative models, such as GANs and Diffusion models, has enabled the creation of highly realistic synthetic images, raising serious concerns about misinformation, deepfakes, and copyright infringement. Although numerous Artificial Intelligence Generated Image (AIGI) detectors have been proposed, often reporting high accuracy, their effectiveness in real-world scenarios remains questionable. To bridge this gap, we introduce AIGIBench, a comprehensive benchmark designed to rigorously evaluate the robustness and generalization capabilities of state-of-the-art AIGI detectors. AIGIBench simulates real-world challenges through four core tasks: multi-source generalization, robustness to image degradation, sensitivity to data augmentation, and impact of test-time pre-processing. It includes 23 diverse fake image subsets that span both advanced and widely adopted image generation techniques, along with real-world samples collected from social media and AI art platforms. Extensive experiments on 11 advanced detectors demonstrate that, despite their high reported accuracy in controlled settings, these detectors suffer significant performance drops on real-world data, limited benefits from common augmentations, and nuanced effects of pre-processing, highlighting the need for more robust detection strategies. By providing a unified and realistic evaluation framework, AIGIBench offers valuable insights to guide future research toward dependable and generalizable AIGI detection.Data and code are publicly available at: https://github.com/HorizonTEL/AIGIBench.
Paper Structure (26 sections, 2 figures, 17 tables)

This paper contains 26 sections, 2 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Training Setting
  3. Evaluation Datasets
  4. Evaluation Metrics
  5. Task Definition
  6. Detection Methods
  7. Experiments and Discussion
  8. Task 1: Generalization Assessment
  9. Task 2: Robustness Assessment
  10. Task 3: Data Augmentation Variation Assessment
  11. Task 4: Test Data Pre-processing Assessment
  12. Conclusions, Limitations, and Board Impacts
  13. Acknowledgments
  14. Appendix
  15. More Results on Task 1: Generalization Assessment
  16. ...and 11 more sections

Figures (2)

  • Figure 1: The AIGI Detection Pipeline. In the training phase, both real and AI-generated images are augmented to improve the model's robustness against diverse and previously unseen test distributions. The model is trained to distinguish real images from synthetic ones generated by a variety of unknown sources. During inference, test images potentially affected by unknown degradations or generation pipelines are pre-processed using cropping or resizing to align with the training conditions. The pre-processed images are then evaluated by the trained detector to assess their authenticity.
  • Figure 2: Visualizations of real and fake images from the evaluation datasets used in our AIGIBench.