Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Audio Deepfake Detection Generalize?

Nicolas M. Müller, Pavel Czempin, Franziska Dieckmann, Adam Froghyar, Konstantin Böttinger

TL;DR

This work interrogates why audio deepfake detectors succeed and whether reported gains generalize beyond lab benchmarks. By re-implementing twelve state-of-the-art architectures under a unified evaluation, the authors isolate the effects of preprocessing, feature choice, and input length, revealing that cqtspec or logspec features outperform melspec by about 37% on average and that full-length audio typically yields better performance than fixed 4-second clips. They also introduce a new in-the-wild dataset with 37.9 hours of real and fake celebrity/politician audio to test generalization, finding serious degradation (up to 1000% increases in EER) compared to ASVspoof benchmarks, suggesting current detectors overfit lab data. The results underscore the need for broader evaluation and more robust features when deploying audio deepfake detection in real-world settings.

Abstract

Current text-to-speech algorithms produce realistic fakes of human voices, making deepfake detection a much-needed area of research. While researchers have presented various techniques for detecting audio spoofs, it is often unclear exactly why these architectures are successful: Preprocessing steps, hyperparameter settings, and the degree of fine-tuning are not consistent across related work. Which factors contribute to success, and which are accidental? In this work, we address this problem: We systematize audio spoofing detection by re-implementing and uniformly evaluating architectures from related work. We identify overarching features for successful audio deepfake detection, such as using cqtspec or logspec features instead of melspec features, which improves performance by 37% EER on average, all other factors constant. Additionally, we evaluate generalization capabilities: We collect and publish a new dataset consisting of 37.9 hours of found audio recordings of celebrities and politicians, of which 17.2 hours are deepfakes. We find that related work performs poorly on such real-world data (performance degradation of up to one thousand percent). This may suggest that the community has tailored its solutions too closely to the prevailing ASVSpoof benchmark and that deepfakes are much harder to detect outside the lab than previously thought.

Does Audio Deepfake Detection Generalize?

TL;DR

This work interrogates why audio deepfake detectors succeed and whether reported gains generalize beyond lab benchmarks. By re-implementing twelve state-of-the-art architectures under a unified evaluation, the authors isolate the effects of preprocessing, feature choice, and input length, revealing that cqtspec or logspec features outperform melspec by about 37% on average and that full-length audio typically yields better performance than fixed 4-second clips. They also introduce a new in-the-wild dataset with 37.9 hours of real and fake celebrity/politician audio to test generalization, finding serious degradation (up to 1000% increases in EER) compared to ASVspoof benchmarks, suggesting current detectors overfit lab data. The results underscore the need for broader evaluation and more robust features when deploying audio deepfake detection in real-world settings.

Abstract

Current text-to-speech algorithms produce realistic fakes of human voices, making deepfake detection a much-needed area of research. While researchers have presented various techniques for detecting audio spoofs, it is often unclear exactly why these architectures are successful: Preprocessing steps, hyperparameter settings, and the degree of fine-tuning are not consistent across related work. Which factors contribute to success, and which are accidental? In this work, we address this problem: We systematize audio spoofing detection by re-implementing and uniformly evaluating architectures from related work. We identify overarching features for successful audio deepfake detection, such as using cqtspec or logspec features instead of melspec features, which improves performance by 37% EER on average, all other factors constant. Additionally, we evaluate generalization capabilities: We collect and publish a new dataset consisting of 37.9 hours of found audio recordings of celebrities and politicians, of which 17.2 hours are deepfakes. We find that related work performs poorly on such real-world data (performance degradation of up to one thousand percent). This may suggest that the community has tailored its solutions too closely to the prevailing ASVSpoof benchmark and that deepfakes are much harder to detect outside the lab than previously thought.
Paper Structure (16 sections, 1 figure, 2 tables)

This paper contains 16 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Architectures
  4. Datasets
  5. Experimental Setup
  6. Training and Evaluation
  7. Hyper Parameters
  8. Train and Evaluation Data Splits
  9. Evaluation metrics
  10. Feature Extraction
  11. Audio Input Length
  12. Results
  13. Fixed vs. Variable Input Length
  14. Effects of Feature Extraction Techniques
  15. Evaluation on 'in-the-wild' data
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Schematics of our collected dataset. For $n=58$ celebrities and politicians, we collected both bona-fide and spoofed audio (represented by blue and red boxes per speaker). In total, we collected $20.8$ hours of bona-fide and $17.2$ hours of spoofed audio. On average, there are $23$ minutes of bona-fide and $18$ minutes of spoofed audio per speaker.