Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deepfake Detection of Singing Voices With Whisper Encodings

Falguni Sharma, Priyanka Gupta

TL;DR

This work tackles singing voice deepfake detection (SVDD) by leveraging noise-variant encodings from the Whisper ASR model as discriminative features. The authors demonstrate that Whisper encodings outperform standard spectral features on both isolated vocals and mixtures, with a ResNet34 classifier achieving the best results across Whisper sizes. A key insight is that Whisper encodings, though not noise-invariant for ASR, carry noise-conditioned information that helps distinguish bonafide from deepfake singing, though unseen languages (T04) still pose challenges. The findings suggest practical improvements for SVDD in music, offering a path toward more robust defense against deepfake singing technologies in real-world applications.

Abstract

The deepfake generation of singing vocals is a concerning issue for artists in the music industry. In this work, we propose a singing voice deepfake detection (SVDD) system, which uses noise-variant encodings of open-AI's Whisper model. As counter-intuitive as it may sound, even though the Whisper model is known to be noise-robust, the encodings are rich in non-speech information, and are noise-variant. This leads us to evaluate Whisper encodings as feature representations for the SVDD task. Therefore, in this work, the SVDD task is performed on vocals and mixtures, and the performance is evaluated in \%EER over varying Whisper model sizes and two classifiers- CNN and ResNet34, under different testing conditions.

Deepfake Detection of Singing Voices With Whisper Encodings

TL;DR

This work tackles singing voice deepfake detection (SVDD) by leveraging noise-variant encodings from the Whisper ASR model as discriminative features. The authors demonstrate that Whisper encodings outperform standard spectral features on both isolated vocals and mixtures, with a ResNet34 classifier achieving the best results across Whisper sizes. A key insight is that Whisper encodings, though not noise-invariant for ASR, carry noise-conditioned information that helps distinguish bonafide from deepfake singing, though unseen languages (T04) still pose challenges. The findings suggest practical improvements for SVDD in music, offering a path toward more robust defense against deepfake singing technologies in real-world applications.

Abstract

The deepfake generation of singing vocals is a concerning issue for artists in the music industry. In this work, we propose a singing voice deepfake detection (SVDD) system, which uses noise-variant encodings of open-AI's Whisper model. As counter-intuitive as it may sound, even though the Whisper model is known to be noise-robust, the encodings are rich in non-speech information, and are noise-variant. This leads us to evaluate Whisper encodings as feature representations for the SVDD task. Therefore, in this work, the SVDD task is performed on vocals and mixtures, and the performance is evaluated in \%EER over varying Whisper model sizes and two classifiers- CNN and ResNet34, under different testing conditions.

Paper Structure

This paper contains 17 sections, 1 equation, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed Work
  3. Whisper Encoder
  4. Noise-Variant Encodings from Noise Robust Whisper
  5. Experimental Setup
  6. Dataset
  7. Classifier and Performance Metrics
  8. Baseline
  9. Experimental Results
  10. Whisper Encodings vs. Standard Representations
  11. Vocals
  12. Mixtures
  13. Analysis on Testing Conditions
  14. Vocals
  15. Mixtures
  16. ...and 2 more sections

Figures (9)

  • Figure 1: The proposed end-to-end SVDD system using Whisper Encoder with ResNet34
  • Figure 2: Bonafide vs. Deepfake Singing Voices
  • Figure 3: %EER for various splits on Singfake vocals across different representations, with ResNet34 as the classifier
  • Figure 4: %EER for various splits on Singfake vocals across different representations, with CNN as the classifier
  • Figure 5: %EER for various splits on Singfake mixtures across different representations, with ResNet34 as the classifier
  • ...and 4 more figures