Multi-Speaker Conversational Audio Deepfake: Taxonomy, Dataset and Pilot Study

TL;DR

The paper addresses the emerging threat of multi-speaker conversational audio deepfakes by proposing a taxonomy of manipulation scopes and introducing MsCADD, a 2,830-clip dataset containing real and fully synthetic two-speaker conversations generated with VITS and SoundStorm. It benchmarks three baselines—LFCC-LCNN, RawNet2, and Wav2Vec 2.0—on MsCADD, reporting accuracy, TPR, TNR, and F1 to establish reference performance. Results show that modern models outperform traditional baselines, yet detecting synthetic voices under varied conversational dynamics remains challenging, particularly in achieving high TPR for fake utterances. The work provides a foundation for future research, including transformer-based detectors and multi-modal approaches, and makes MsCADD publicly available to accelerate research in robust multi-speaker deepfake detection.

Abstract

The rapid advances in text-to-speech (TTS) technologies have made audio deepfakes increasingly realistic and accessible, raising significant security and trust concerns. While existing research has largely focused on detecting single-speaker audio deepfakes, real-world malicious applications with multi-speaker conversational settings is also emerging as a major underexplored threat. To address this gap, we propose a conceptual taxonomy of multi-speaker conversational audio deepfakes, distinguishing between partial manipulations (one or multiple speakers altered) and full manipulations (entire conversations synthesized). As a first step, we introduce a new Multi-speaker Conversational Audio Deepfakes Dataset (MsCADD) of 2,830 audio clips containing real and fully synthetic two-speaker conversations, generated using VITS and SoundStorm-based NotebookLM models to simulate natural dialogue with variations in speaker gender, and conversational spontaneity. MsCADD is limited to text-to-speech (TTS) types of deepfake. We benchmark three neural baseline models; LFCC-LCNN, RawNet2, and Wav2Vec 2.0 on this dataset and report performance in terms of F1 score, accuracy, true positive rate (TPR), and true negative rate (TNR). Results show that these baseline models provided a useful benchmark, however, the results also highlight that there is a significant gap in multi-speaker deepfake research in reliably detecting synthetic voices under varied conversational dynamics. Our dataset and benchmarks provide a foundation for future research on deepfake detection in conversational scenarios, which is a highly underexplored area of research but also a major area of threat to trustworthy information in audio settings. The MsCADD dataset is publicly available to support reproducibility and benchmarking by the research community.

Figures (1)

• Figure 1: Taxonomy of Multi-Speaker Conversational Audio Deepfakes. This structure organizes deepfake scenarios based on interaction context, speaker composition, and manipulation scope.