Investigating Prosodic Signatures via Speech Pre-Trained Models for Audio Deepfake Source Attribution

TL;DR

This work tackles audio deepfake source attribution (ADSD) by evaluating a suite of state-of-the-art speech pre-trained models for their ability to capture source-specific prosodic signatures. It shows that the x-vector model, pretrained for speaker recognition, best captures these signatures among single PTMs, while multilingual and large-scale PTMs also provide useful signals. To exploit complementary information across PTMs, the authors propose FINDER, a Renyi-divergence-based fusion framework that aligns representations from different PTMs; in particular, fusion of Whisper and x-vector via FINDER achieves state-of-the-art performance on ASVSpoof 2019 and CFAD datasets. Overall, the study demonstrates that PTM fusion can substantially improve ADSD performance, enabling finer-grained attribution of synthetic audio sources and suggesting practical utility for forensic and security applications.

Abstract

In this work, we investigate various state-of-the-art (SOTA) speech pre-trained models (PTMs) for their capability to capture prosodic signatures of the generative sources for audio deepfake source attribution (ADSD). These prosodic characteristics can be considered one of major signatures for ADSD, which is unique to each source. So better is the PTM at capturing prosodic signs better the ADSD performance. We consider various SOTA PTMs that have shown top performance in different prosodic tasks for our experiments on benchmark datasets, ASVSpoof 2019 and CFAD. x-vector (speaker recognition PTM) attains the highest performance in comparison to all the PTMs considered despite consisting lowest model parameters. This higher performance can be due to its speaker recognition pre-training that enables it for capturing unique prosodic characteristics of the sources in a better way. Further, motivated from tasks such as audio deepfake detection and speech recognition, where fusion of PTMs representations lead to improved performance, we explore the same and propose FINDER for effective fusion of such representations. With fusion of Whisper and x-vector representations through FINDER, we achieved the topmost performance in comparison to all the individual PTMs as well as baseline fusion techniques and attaining SOTA performance.

Figures (3)

• Figure 1: Proposed Framework FINDER: RD and FCN stand for renyi divergence and fully connected network, respectively; $L$, $L_{CE}$, and $L_{RD}$ represent the total loss, cross-entropy loss, and renyi divergence loss, respectively.
• Figure 2: Representation Space Visualization of PTMs for CFAD
• Figure 3: Representation Space Visualization of PTMs for ASV