Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards generalisable and calibrated synthetic speech detection with self-supervised representations

Octavian Pascu, Adriana Stan, Dan Oneata, Elisabeta Oneata, Horia Cucu

TL;DR

It is shown that large frozen representations coupled with a simple logistic regression classifier are extremely effective in achieving strong generalisation capabilities: compared to the RawNet2 model, this approach reduces the equal error rate from 30.9% to 8.8% on a benchmark of eight deepfake datasets, while learning less than 2k parameters.

Abstract

Generalisation -- the ability of a model to perform well on unseen data -- is crucial for building reliable deepfake detectors. However, recent studies have shown that the current audio deepfake models fall short of this desideratum. In this work we investigate the potential of pretrained self-supervised representations in building general and calibrated audio deepfake detection models. We show that large frozen representations coupled with a simple logistic regression classifier are extremely effective in achieving strong generalisation capabilities: compared to the RawNet2 model, this approach reduces the equal error rate from 30.9% to 8.8% on a benchmark of eight deepfake datasets, while learning less than 2k parameters. Moreover, the proposed method produces considerably more reliable predictions compared to previous approaches making it more suitable for realistic use.

Towards generalisable and calibrated synthetic speech detection with self-supervised representations

TL;DR

It is shown that large frozen representations coupled with a simple logistic regression classifier are extremely effective in achieving strong generalisation capabilities: compared to the RawNet2 model, this approach reduces the equal error rate from 30.9% to 8.8% on a benchmark of eight deepfake datasets, while learning less than 2k parameters.

Abstract

Generalisation -- the ability of a model to perform well on unseen data -- is crucial for building reliable deepfake detectors. However, recent studies have shown that the current audio deepfake models fall short of this desideratum. In this work we investigate the potential of pretrained self-supervised representations in building general and calibrated audio deepfake detection models. We show that large frozen representations coupled with a simple logistic regression classifier are extremely effective in achieving strong generalisation capabilities: compared to the RawNet2 model, this approach reduces the equal error rate from 30.9% to 8.8% on a benchmark of eight deepfake datasets, while learning less than 2k parameters. Moreover, the proposed method produces considerably more reliable predictions compared to previous approaches making it more suitable for realistic use.
Paper Structure (11 sections, 3 figures, 4 tables)

This paper contains 11 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experimental setup
  4. Experimental results
  5. How well do self-supervised representations generalise?
  6. How reliable are the self-supervised representations?
  7. How do other self-supervised representations perform?
  8. How important is the classifier?
  9. What is the impact of training data?
  10. Conclusions
  11. Acknowledgements

Figures (3)

  • Figure 1: Evaluation of reliability estimation in terms of accuracy and fraction of samples kept, as we vary the reliability threshold $\tau \in [0, 1]$. Our results (blue) are more reliable than those of Salvi et al.salvi2023icassp (orange) on both metrics and datasets.
  • Figure 2: Performance trade off as a function of inference time for the 11 variants of self-supervised representations. Marker area indicates peak video memory. Time and memory are averaged over 64 audio files, which average three seconds.
  • Figure 3: Performance versus number of ASVspoof'19 training samples using wav2vec2/xls-r-2b. Error bars are one standard deviation over three random subsets of training data.