Anomaly Detection and Localization for Speech Deepfakes via Feature Pyramid Matching
Emma Coletta, Davide Salvi, Viola Negroni, Daniele Ugo Leonzio, Paolo Bestagini
TL;DR
This work tackles the problem of robust speech deepfake detection under generalization and interpretability constraints. It reframes detection as one-class anomaly detection by training exclusively on real speech, modeling $p_{real}$, and flagging outliers as fake, while delivering time–frequency anomaly maps. The approach employs a Student-Teacher Feature Pyramid Matching framework augmented with Discrepancy Scaling to improve generalization to unseen generators and to provide calibrated anomaly scores. Through extensive single- and multi-speaker experiments on diverse datasets, the method shows superior generalization over supervised baselines and offers interpretable localization of synthetic artifacts in the audio signal, facilitating forensic analysis and real-world deployment.
Abstract
The rise of AI-driven generative models has enabled the creation of highly realistic speech deepfakes - synthetic audio signals that can imitate target speakers' voices - raising critical security concerns. Existing methods for detecting speech deepfakes primarily rely on supervised learning, which suffers from two critical limitations: limited generalization to unseen synthesis techniques and a lack of explainability. In this paper, we address these issues by introducing a novel interpretable one-class detection framework, which reframes speech deepfake detection as an anomaly detection task. Our model is trained exclusively on real speech to characterize its distribution, enabling the classification of out-of-distribution samples as synthetically generated. Additionally, our framework produces interpretable anomaly maps during inference, highlighting anomalous regions across both time and frequency domains. This is done through a Student-Teacher Feature Pyramid Matching system, enhanced with Discrepancy Scaling to improve generalization capabilities across unseen data distributions. Extensive evaluations demonstrate the superior performance of our approach compared to the considered baselines, validating the effectiveness of framing speech deepfake detection as an anomaly detection problem.