Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Deepart Detection

Yabin Wang, Zhiwu Huang, Xiaopeng Hong

TL;DR

This paper introduces the first deepart detection benchmark, the Deepart Detection Database (DDDB), pairing LAION-5B conarts with deeparts generated by five state-of-the-art models to study deepart detection. It defines two novel problems—once-for-all deepart detection (ODD) and continual deepart detection (CDD)—and proposes four benchmark evaluations plus four solution families. The authors show that continual learning approaches generally outperform isolated ODD methods and introduce a transformation framework based on Knowledge Distillation, Cosine Normalization, and Prompt-Tuning to rescue collapsed rehearsal-free methods in the most challenging CDD3 setting. The work provides extensive experiments with ViT backbones, data augmentation, and a copyright-identification metric, offering practical guidance for real-world deepart copyright protection and detection research.

Abstract

Deepfake technologies have been blurring the boundaries between the real and unreal, likely resulting in malicious events. By leveraging newly emerged deepfake technologies, deepfake researchers have been making a great upending to create deepfake artworks (deeparts), which are further closing the gap between reality and fantasy. To address potentially appeared ethics questions, this paper establishes a deepart detection database (DDDB) that consists of a set of high-quality conventional art images (conarts) and five sets of deepart images generated by five state-of-the-art deepfake models. This database enables us to explore once-for-all deepart detection and continual deepart detection. For the two new problems, we suggest four benchmark evaluations and four families of solutions on the constructed DDDB. The comprehensive study demonstrates the effectiveness of the proposed solutions on the established benchmark dataset, which is capable of paving a way to more interesting directions of deepart detection. The constructed benchmark dataset and the source code will be made publicly available.

Benchmarking Deepart Detection

TL;DR

This paper introduces the first deepart detection benchmark, the Deepart Detection Database (DDDB), pairing LAION-5B conarts with deeparts generated by five state-of-the-art models to study deepart detection. It defines two novel problems—once-for-all deepart detection (ODD) and continual deepart detection (CDD)—and proposes four benchmark evaluations plus four solution families. The authors show that continual learning approaches generally outperform isolated ODD methods and introduce a transformation framework based on Knowledge Distillation, Cosine Normalization, and Prompt-Tuning to rescue collapsed rehearsal-free methods in the most challenging CDD3 setting. The work provides extensive experiments with ViT backbones, data augmentation, and a copyright-identification metric, offering practical guidance for real-world deepart copyright protection and detection research.

Abstract

Deepfake technologies have been blurring the boundaries between the real and unreal, likely resulting in malicious events. By leveraging newly emerged deepfake technologies, deepfake researchers have been making a great upending to create deepfake artworks (deeparts), which are further closing the gap between reality and fantasy. To address potentially appeared ethics questions, this paper establishes a deepart detection database (DDDB) that consists of a set of high-quality conventional art images (conarts) and five sets of deepart images generated by five state-of-the-art deepfake models. This database enables us to explore once-for-all deepart detection and continual deepart detection. For the two new problems, we suggest four benchmark evaluations and four families of solutions on the constructed DDDB. The comprehensive study demonstrates the effectiveness of the proposed solutions on the established benchmark dataset, which is capable of paving a way to more interesting directions of deepart detection. The constructed benchmark dataset and the source code will be made publicly available.
Paper Structure (16 sections, 5 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 16 sections, 5 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Database
  4. Database Construction
  5. Probed Problems
  6. Benchmark Evaluations
  7. Suggested Methods
  8. Revisiting State-of-the-art Methods for ODD
  9. Adapting Previous Methods for CDD1/CDD2
  10. Rescuing Collapsed/Failed Methods for CDD3
  11. Experiments
  12. Conclusions and Discussions
  13. Computation of Suggested Benchmark Evaluation Metrics
  14. Detailed Algorithm of Proposed Transformation Framework on Rehearsal-free Methods for CDD3
  15. Training Details of Suggested/Transformed Methods
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Examples of the established deepart detection database (DDDB). The examples of LAION-5B schuhmann2022laion are conventional artworks (conarts), and the rest examples (i.e., StableDiff rombach2021highresolution,DALL-E 2 ramesh2022hierarchical,Imagen saharia2022photorealistic,Midjourney david2022mj, and Parti yu2022scaling) are deepfake artworks (deeparts) produced by generative models. Our data and code will be released.
  • Figure 2: Illustration of the proposed framework to rescue the rehearsal-free methods (a) LwF li2017learning, (b) EWC kirkpatrick2017overcoming and (c) S-Prompts wang2022sprompt using the three suggested techniques: (1) Knowledge Distillation (KD), (2) Cosine Normalization (CN), and (3) Prompt-Tuning (PT).
  • Figure 3: Frequency analysis on each deepfake/deepart model generated images. The periodic patterns (dots or lines) of our used deepart models (StableDiff, DALL-E2, Parti, Midjourney, Imagen) generated deepfake art images are highly close to those of conarts from LAION-5B and the real images from ImageNet. In contrast, images generated by early deepfake models (ProGAN karras2018progressive,GauGAN park2019SPADE, BigGAN brock2018large, CycleGAN zhu2017unpaired, StarGAN choi2018stargan) have different periodic patterns.
  • Figure 4: t-SNE visualization of learned feature spaces of LwF, EWC and S-Prompts for CDD3. (a), (b), (c): without using our proposed transformation framework, (d), (e), (f): using our framework.