Learning on Less: Constraining Pre-trained Model Learning for Generalizable Diffusion-Generated Image Detection

TL;DR

This paper tackles the challenge of detecting diffusion-generated images with strong generalization to unseen models. It introduces Learning on Less (LoL), a framework that leverages pre-trained weights and enforces learning constraints via random masking to suppress diffusion-model-specific patterns, guiding the model toward a universal real-vs-fake distinction. The authors formalize the objective around an optimal generalization solution $\theta^{*}$ and show that pre-trained encoders can approximate this solution at certain training steps, though with instability; masking stabilizes learning and improves generalization. Empirical results on GenImage demonstrate state-of-the-art performance, achieving an AvgAcc improvement of up to 13.6% with only 1% of training data and strong robustness across eight unseen diffusion-model generators, highlighting practical impact for misinformation mitigation with limited data. Key contributions include the problem formulation, the random mask generation algorithm, and extensive ablations, establishing LoL as a scalable approach to universal diffusion-generated image detection.

Abstract

Diffusion Models enable realistic image generation, raising the risk of misinformation and eroding public trust. Currently, detecting images generated by unseen diffusion models remains challenging due to the limited generalization capabilities of existing methods. To address this issue, we rethink the effectiveness of pre-trained models trained on large-scale, real-world images. Our findings indicate that: 1) Pre-trained models can cluster the features of real images effectively. 2) Models with pre-trained weights can approximate an optimal generalization solution at a specific training step, but it is extremely unstable. Based on these facts, we propose a simple yet effective training method called Learning on Less (LoL). LoL utilizes a random masking mechanism to constrain the model's learning of the unique patterns specific to a certain type of diffusion model, allowing it to focus on less image content. This leverages the inherent strengths of pre-trained weights while enabling a more stable approach to optimal generalization, which results in the extraction of a universal feature that differentiates various diffusion-generated images from real images. Extensive experiments on the GenImage benchmark demonstrate the remarkable generalization capability of our proposed LoL. With just 1% training data, LoL significantly outperforms the current state-of-the-art, achieving a 13.6% improvement in average ACC across images generated by eight different models.

Figures (9)

• Figure 1: t-SNE visualization van2008visualizing of features from images generated by eight models in the GenImage dataset. (a) and (d) Feature space of a standard ResNet50 and our proposed method trained on images generated by a single diffusion model type (ADM). (b) Feature space of a standard ResNet50 trained on images generated by all involved model types. (c) Feature space of a zero-shot pre-trained CLIP-ResNet50.
• Figure 2: Average Accuracy Across GenImage Test Sets for Models Trained on ADM. Performance comparison at various training steps for Normal ResNet50, pre-trained CLIP-ResNet50, and pre-trained CLIP-ResNet50 using our proposed method.
• Figure 3: Overview of the Constrained Learning Process Pipeline.
• Figure 4: Accuracy (ACC) Results Across 8 Subsets. Each model is trained on a single subset and evaluated across all 8 subsets. The comparison includes the standard ResNet50 he2016deep, $\text{LaRE}^{2}$luo2024lare, and our proposed method. The color scale reflects performance, with darker shades representing higher accuracy values.
• Figure 5: Average Precision (AP) Results Across 8 Subsets. Evaluation of the standard ResNet50, $\text{LaRE}^{2}$, and our proposed method across all 8 subsets, performed using the same approach as the accuracy (ACC) assessment.