Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust AI-Generated Face Detection with Imbalanced Data

Yamini Sri Krubha, Aryana Hou, Braden Vester, Web Walker, Xin Wang, Li Lin, Shu Hu

TL;DR

This work tackles robust detection of AI-generated face media in the presence of severe data imbalance and evolving generation techniques. It introduces a CLIP-based detection framework that fuses a frozen CLIP ViT-L/14 encoder with a 3-layer MLP, enhanced by latent feature augmentation and a learning objective that combines CVaR loss and AUC loss, optimized via sharpness-aware minimization. Empirical results on the DFWild-Cup benchmark show state-of-the-art performance across AUC, accuracy, F1, and EER, with ablations confirming the value of CVaR for imbalanced data and the effectiveness of a compact MLP. The approach offers robust cross-domain detection with strong generalization to unseen generative models, making it well-suited for real-world deployment where data imbalance and model evolution are persistent concerns.

Abstract

Deepfakes, created using advanced AI techniques such as Variational Autoencoder and Generative Adversarial Networks, have evolved from research and entertainment applications into tools for malicious activities, posing significant threats to digital trust. Current deepfake detection techniques have evolved from CNN-based methods focused on local artifacts to more advanced approaches using vision transformers and multimodal models like CLIP, which capture global anomalies and improve cross-domain generalization. Despite recent progress, state-of-the-art deepfake detectors still face major challenges in handling distribution shifts from emerging generative models and addressing severe class imbalance between authentic and fake samples in deepfake datasets, which limits their robustness and detection accuracy. To address these challenges, we propose a framework that combines dynamic loss reweighting and ranking-based optimization, which achieves superior generalization and performance under imbalanced dataset conditions. The code is available at https://github.com/Purdue-M2/SP_CUP.

Robust AI-Generated Face Detection with Imbalanced Data

TL;DR

This work tackles robust detection of AI-generated face media in the presence of severe data imbalance and evolving generation techniques. It introduces a CLIP-based detection framework that fuses a frozen CLIP ViT-L/14 encoder with a 3-layer MLP, enhanced by latent feature augmentation and a learning objective that combines CVaR loss and AUC loss, optimized via sharpness-aware minimization. Empirical results on the DFWild-Cup benchmark show state-of-the-art performance across AUC, accuracy, F1, and EER, with ablations confirming the value of CVaR for imbalanced data and the effectiveness of a compact MLP. The approach offers robust cross-domain detection with strong generalization to unseen generative models, making it well-suited for real-world deployment where data imbalance and model evolution are persistent concerns.

Abstract

Deepfakes, created using advanced AI techniques such as Variational Autoencoder and Generative Adversarial Networks, have evolved from research and entertainment applications into tools for malicious activities, posing significant threats to digital trust. Current deepfake detection techniques have evolved from CNN-based methods focused on local artifacts to more advanced approaches using vision transformers and multimodal models like CLIP, which capture global anomalies and improve cross-domain generalization. Despite recent progress, state-of-the-art deepfake detectors still face major challenges in handling distribution shifts from emerging generative models and addressing severe class imbalance between authentic and fake samples in deepfake datasets, which limits their robustness and detection accuracy. To address these challenges, we propose a framework that combines dynamic loss reweighting and ranking-based optimization, which achieves superior generalization and performance under imbalanced dataset conditions. The code is available at https://github.com/Purdue-M2/SP_CUP.
Paper Structure (22 sections, 5 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 22 sections, 5 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deepfake Generation
  4. Deepfake Detection
  5. Robust Deepfake Detection
  6. Method
  7. Overview
  8. Pre-trained Models
  9. Training Data Augmentation
  10. Learning Objective
  11. Optimization
  12. Algorithm
  13. Experiments
  14. Experimental Settings
  15. Datasets
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Overview of our proposed model using CLIP ViT-L/14 for encoding input images and an MLP module trained with a robust loss function combining CVaR on visual similarity and weighted AUC. The optimization process leverages a flattened loss landscape to ensure robust deepfake detection between real and fake images.
  • Figure 2: Visualization of the ROC curve (a) and training dynamics (b) for our proposed method.