Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comprehensive Layer-wise Analysis of SSL Models for Audio Deepfake Detection

Yassine El Kheir, Youness Samih, Suraj Maharjan, Tim Polzehl, Sebastian Möller

TL;DR

This study addresses the challenge of audio deepfake detection across multilingual and diverse contextual scenarios by performing a layer-wise analysis of self-supervised learning (SSL) front-ends (Wav2Vec2, Hubert, WavLM) and two back-ends (FFN, AASIST). By freezing the SSL front-ends and learning layer-wise weights, it reveals that lower transformer layers consistently provide the most discriminative features, enabling reduced-layer configurations (4-6 for small models; 10-12 for large models) to achieve competitive or superior performance while lowering computational cost. The findings generalize across full, partial, song, and scene deepfakes and across English, Chinese, and Spanish datasets, highlighting a focus on local features and artifacts in early layers. The work provides practical guidance for efficient, multilingual SSL-based deepfake detectors and offers publicly available models and code for replication and extension.

Abstract

This paper conducts a comprehensive layer-wise analysis of self-supervised learning (SSL) models for audio deepfake detection across diverse contexts, including multilingual datasets (English, Chinese, Spanish), partial, song, and scene-based deepfake scenarios. By systematically evaluating the contributions of different transformer layers, we uncover critical insights into model behavior and performance. Our findings reveal that lower layers consistently provide the most discriminative features, while higher layers capture less relevant information. Notably, all models achieve competitive equal error rate (EER) scores even when employing a reduced number of layers. This indicates that we can reduce computational costs and increase the inference speed of detecting deepfakes by utilizing only a few lower layers. This work enhances our understanding of SSL models in deepfake detection, offering valuable insights applicable across varied linguistic and contextual settings. Our trained models and code are publicly available: https://github.com/Yaselley/SSL_Layerwise_Deepfake.

Comprehensive Layer-wise Analysis of SSL Models for Audio Deepfake Detection

TL;DR

This study addresses the challenge of audio deepfake detection across multilingual and diverse contextual scenarios by performing a layer-wise analysis of self-supervised learning (SSL) front-ends (Wav2Vec2, Hubert, WavLM) and two back-ends (FFN, AASIST). By freezing the SSL front-ends and learning layer-wise weights, it reveals that lower transformer layers consistently provide the most discriminative features, enabling reduced-layer configurations (4-6 for small models; 10-12 for large models) to achieve competitive or superior performance while lowering computational cost. The findings generalize across full, partial, song, and scene deepfakes and across English, Chinese, and Spanish datasets, highlighting a focus on local features and artifacts in early layers. The work provides practical guidance for efficient, multilingual SSL-based deepfake detectors and offers publicly available models and code for replication and extension.

Abstract

This paper conducts a comprehensive layer-wise analysis of self-supervised learning (SSL) models for audio deepfake detection across diverse contexts, including multilingual datasets (English, Chinese, Spanish), partial, song, and scene-based deepfake scenarios. By systematically evaluating the contributions of different transformer layers, we uncover critical insights into model behavior and performance. Our findings reveal that lower layers consistently provide the most discriminative features, while higher layers capture less relevant information. Notably, all models achieve competitive equal error rate (EER) scores even when employing a reduced number of layers. This indicates that we can reduce computational costs and increase the inference speed of detecting deepfakes by utilizing only a few lower layers. This work enhances our understanding of SSL models in deepfake detection, offering valuable insights applicable across varied linguistic and contextual settings. Our trained models and code are publicly available: https://github.com/Yaselley/SSL_Layerwise_Deepfake.

Paper Structure

This paper contains 35 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Front-Ends: SSL Models
  5. Wav2Vec2.0
  6. Hubert
  7. WavLM
  8. Model Selection and Multilingual Setup
  9. Back-End Classifiers: FFN vs AASIST
  10. FFN
  11. AASIST
  12. Experimental Design
  13. Datasets
  14. Training setup
  15. Results and Discussion
  16. ...and 20 more sections

Figures (3)

  • Figure 1: Layer-wise Contribution Framework. The framework consists of SSL models as front-ends to extract features and a back-end classifier. The front-end SSL models remain frozen during the experiments to evaluate the layer-wise feature contribution.
  • Figure 2: Heatmap of Normalized Layer-wise weights Across Various Datasets using Small SSL Models. AVERAGE row representing the average weights across datasets.
  • Figure 3: Heatmap of Normalized Layer-wise weights Across Various Datasets using Large SSL Models. AVERAGE row representing the average weights across datasets.