ImagiNet: A Multi-Content Benchmark for Synthetic Image Detection

TL;DR

This work tackles the generalization gap in synthetic-image detection by introducing ImagiNet, a high-resolution, multi-content benchmark spanning photos, paintings, faces, and miscellaneous images, with synthetic content drawn from open-source and proprietary generators. It employs a two-track evaluation: real vs synthetic detection and generator identification, under perturbations like JPEG compression and resizing. A strong baseline based on a ResNet-50 backbone trained with the self-supervised contrastive objective $\mathcal{L}_{SC}$ achieves up to $0.99$ AUC and robust accuracy, and demonstrates zero-shot generalization to prior benchmarks, highlighting the value of content-type diversity. The dataset and code enable robust detector development and offer insights into how balanced, diverse training data improves generalization in synthetic-content detection across diverse generators and content types.

Abstract

Recent generative models produce images with a level of authenticity that makes them nearly indistinguishable from real photos and artwork. Potential harmful use cases of these models, necessitate the creation of robust synthetic image detectors. However, current datasets in the field contain generated images with questionable quality or have examples from one predominant content type which leads to poor generalizability of the underlying detectors. We find that the curation of a balanced amount of high-resolution generated images across various content types is crucial for the generalizability of detectors, and introduce ImagiNet, a dataset of 200K examples, spanning four categories: photos, paintings, faces, and miscellaneous. Synthetic images in ImagiNet are produced with both open-source and proprietary generators, whereas real counterparts for each content type are collected from public datasets. The structure of ImagiNet allows for a two-track evaluation system: i) classification as real or synthetic and ii) identification of the generative model. To establish a strong baseline, we train a ResNet-50 model using a self-supervised contrastive objective (SelfCon) for each track which achieves evaluation AUC of up to 0.99 and balanced accuracy ranging from 86% to 95%, even under conditions that involve compression and resizing. The provided model is generalizable enough to achieve zero-shot state-of-the-art performance on previous synthetic detection benchmarks. We provide ablations to demonstrate the importance of content types and publish code and data.

Figures (5)

• Figure 1: Prompt structures for image generation.
• Figure 2: Dimensionality reduction vizualization of the backbone representations for a subset of ImagiNet.
• Figure 3: Mean accuracy and AUC on the different models trained by leaving one content type out.
• Figure 4: Accuracy of model identification classifier under perturbations.
• Figure 5: Positive suffixes (green) and negative prompts (red) utilized for the generation of all generative models requiring prompts.