Harder or Different? Understanding Generalization of Audio Deepfake Detection

TL;DR

This work addresses the generalization problem in audio deepfake detection by decomposing the performance gap between in-domain and out-of-domain data into two components: a hardness gap reflecting inherent difficulty and a difference gap reflecting distribution mismatch. Using four detectors and ASVspoof 2019/2021 datasets plus In-The-Wild data, the authors show that the difference gap is the dominant contributor to generalization failure, especially for unseen attacks, while increasing model capacity yields limited gains for cross-domain generalization. Self-supervised models (e.g., SSL-W2V2, WhisperDF) tend to exhibit smaller gaps, suggesting that less attack-specific representations improve transfer. The findings imply that practical defenses should focus on mitigating cross-domain attack differences and distribution shifts rather than solely expanding model capacity, and call for broader analyses of augmentations and attack families. These insights have significant implications for deploying robust, real-world audio deepfake detectors.

Abstract

Recent research has highlighted a key issue in speech deepfake detection: models trained on one set of deepfakes perform poorly on others. The question arises: is this due to the continuously improving quality of Text-to-Speech (TTS) models, i.e., are newer DeepFakes just 'harder' to detect? Or, is it because deepfakes generated with one model are fundamentally different to those generated using another model? We answer this question by decomposing the performance gap between in-domain and out-of-domain test data into 'hardness' and 'difference' components. Experiments performed using ASVspoof databases indicate that the hardness component is practically negligible, with the performance gap being attributed primarily to the difference component. This has direct implications for real-world deepfake detection, highlighting that merely increasing model capacity, the currently-dominant research trend, may not effectively address the generalization challenge.

Figures (1)

• Figure 1: Comparison of EER between ASVspoof 2019 (x-Axis) and ASVspoof 2021 (y-axis) for models trained on ASVspoof 2019. None of the model achieves ideal generalization, as denoted by the dashed line.