Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Data

Aref Azizpour, Tai D. Nguyen, Manil Shrestha, Kaidi Xu, Edward Kim, Matthew C. Stamm

TL;DR

The Ensemble of Expert Embedders (E3) is introduced, a novel continual learning framework for updating synthetic image detectors that enables the accurate detection of images from newly emerged generators using minimal training data.

Abstract

As generative AI progresses rapidly, new synthetic image generators continue to emerge at a swift pace. Traditional detection methods face two main challenges in adapting to these generators: the forensic traces of synthetic images from new techniques can vastly differ from those learned during training, and access to data for these new generators is often limited. To address these issues, we introduce the Ensemble of Expert Embedders (E3), a novel continual learning framework for updating synthetic image detectors. E3 enables the accurate detection of images from newly emerged generators using minimal training data. Our approach does this by first employing transfer learning to develop a suite of expert embedders, each specializing in the forensic traces of a specific generator. Then, all embeddings are jointly analyzed by an Expert Knowledge Fusion Network to produce accurate and reliable detection decisions. Our experiments demonstrate that E3 outperforms existing continual learning methods, including those developed specifically for synthetic image detection.

E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Data

TL;DR

The Ensemble of Expert Embedders (E3) is introduced, a novel continual learning framework for updating synthetic image detectors that enables the accurate detection of images from newly emerged generators using minimal training data.

Abstract

As generative AI progresses rapidly, new synthetic image generators continue to emerge at a swift pace. Traditional detection methods face two main challenges in adapting to these generators: the forensic traces of synthetic images from new techniques can vastly differ from those learned during training, and access to data for these new generators is often limited. To address these issues, we introduce the Ensemble of Expert Embedders (E3), a novel continual learning framework for updating synthetic image detectors. E3 enables the accurate detection of images from newly emerged generators using minimal training data. Our approach does this by first employing transfer learning to develop a suite of expert embedders, each specializing in the forensic traces of a specific generator. Then, all embeddings are jointly analyzed by an Expert Knowledge Fusion Network to produce accurate and reliable detection decisions. Our experiments demonstrate that E3 outperforms existing continual learning methods, including those developed specifically for synthetic image detection.
Paper Structure (16 sections, 3 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 16 sections, 3 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Formulation
  4. Proposed Approach
  5. Creating A New Expert Embedder
  6. Expert Knowledge Fusion Network
  7. E3 Framework Summary
  8. Experiments
  9. Experimental Setup
  10. Adapting to One New Generator
  11. Adapting to Multiple Emerging Generators
  12. Effects of New Generator's Training Data Size
  13. Effects of Different Detector Architectures
  14. Discussion
  15. Ablation Results
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Visualization of the Ensemble of Expert Embedders (E3) framework aimed at enhancing synthetic image detection in response to new generators. Examples of the forensic residual traces for different generator architectures are also depicted.
  • Figure 2: End-to-end architecture workflow: An image is first processed by E3 to generate embeddings, which are passed through a transformer and MLP layers to produce the detection decision.
  • Figure 3: Creation of a new specialized expert, $\phi_k$, when a new generator emerges. The baseline detector $f_0$ is fine-tuned with data $\mathcal{D}_k$ and real images $\mathcal{R}$ from memory buffer. The new classifier $\hat{f}_k$'s embedder $\phi_k$ is then preserved.
  • Figure 4: Our method shows slight performance decline when adapting to 19 generators with training image counts reduced from 500 to 50 per generator.