Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis

Huy H. Nguyen, Junichi Yamagishi, Isao Echizen

TL;DR

This work evaluates self-supervised vision transformers (ViTs) for facial deepfake detection, comparing frozen backbones and partial fine-tuning against supervised ViTs and ConvNets. It shows that SSL pre-training (notably DINOv2 and MAE) yields strong detection performance and attention-based explainability with modest data, while partially fine-tuning the final transformer blocks provides the best accuracy and interpretability. Cross-dataset experiments demonstrate improved generalization for SSL backbones, particularly DINOv2, though unseen diffusion-based methods remain challenging. The findings advocate for SSL ViTs as resource-efficient, interpretable backbones for robust deepfake detection and motivate future localization and unlabeled-data exploration in forensics.

Abstract

This paper investigates the effectiveness of self-supervised pre-trained vision transformers (ViTs) compared to supervised pre-trained ViTs and conventional neural networks (ConvNets) for detecting facial deepfake images and videos. It examines their potential for improved generalization and explainability, especially with limited training data. Despite the success of transformer architectures in various tasks, the deepfake detection community is hesitant to use large ViTs as feature extractors due to their perceived need for extensive data and suboptimal generalization with small datasets. This contrasts with ConvNets, which are already established as robust feature extractors. Additionally, training ViTs from scratch requires significant resources, limiting their use to large companies. Recent advancements in self-supervised learning (SSL) for ViTs, like masked autoencoders and DINOs, show adaptability across diverse tasks and semantic segmentation capabilities. By leveraging SSL ViTs for deepfake detection with modest data and partial fine-tuning, we find comparable adaptability to deepfake detection and explainability via the attention mechanism. Moreover, partial fine-tuning of ViTs is a resource-efficient option.

Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis

TL;DR

This work evaluates self-supervised vision transformers (ViTs) for facial deepfake detection, comparing frozen backbones and partial fine-tuning against supervised ViTs and ConvNets. It shows that SSL pre-training (notably DINOv2 and MAE) yields strong detection performance and attention-based explainability with modest data, while partially fine-tuning the final transformer blocks provides the best accuracy and interpretability. Cross-dataset experiments demonstrate improved generalization for SSL backbones, particularly DINOv2, though unseen diffusion-based methods remain challenging. The findings advocate for SSL ViTs as resource-efficient, interpretable backbones for robust deepfake detection and motivate future localization and unlabeled-data exploration in forensics.

Abstract

This paper investigates the effectiveness of self-supervised pre-trained vision transformers (ViTs) compared to supervised pre-trained ViTs and conventional neural networks (ConvNets) for detecting facial deepfake images and videos. It examines their potential for improved generalization and explainability, especially with limited training data. Despite the success of transformer architectures in various tasks, the deepfake detection community is hesitant to use large ViTs as feature extractors due to their perceived need for extensive data and suboptimal generalization with small datasets. This contrasts with ConvNets, which are already established as robust feature extractors. Additionally, training ViTs from scratch requires significant resources, limiting their use to large companies. Recent advancements in self-supervised learning (SSL) for ViTs, like masked autoencoders and DINOs, show adaptability across diverse tasks and semantic segmentation capabilities. By leveraging SSL ViTs for deepfake detection with modest data and partial fine-tuning, we find comparable adaptability to deepfake detection and explainability via the attention mechanism. Moreover, partial fine-tuning of ViTs is a resource-efficient option.
Paper Structure (18 sections, 3 equations, 3 figures, 7 tables)

This paper contains 18 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-supervised vision transformers
  4. Transformers in deepfake detection
  5. Methodology
  6. Problem formalization
  7. Approach 1: Using frozen backbone as a multi-level feature extractor
  8. Approach 2: Fine-tuning last transformer blocks
  9. Experimental Design
  10. Backbones
  11. Datasets
  12. Metrics
  13. Results and discussion
  14. Approach 1: Using frozen backbone as a multi-level feature extractor
  15. Approach 2: Fine-tuning final transformer blocks
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Overview of the two investigated approaches. Blue blocks mean frozen blocks, while orange blocks mean fine-tuned or trained blocks.
  • Figure 2: Ablation study on the relationship between the number of fine-tuned transformer blocks ($k$) and the EER in Approach 2.
  • Figure 3: Visualization of averages of final block's multi-head attention maps of the partially fine-tuned DINOv2 - ViT-L/14-Reg from Approach 2, compared with those from its original pre-trained version. The training dataset includes Deepfakes and FaceSwap, while P2 and Repaint-LDM are unseen methods. Best viewed in color.