ASVspoof 5: Evaluation of Spoofing, Deepfake, and Adversarial Attack Detection Using Crowdsourced Speech

TL;DR

ASVspoof 5 introduces a crowdsourced MLS English dataset with ~2k speakers to evaluate spoof/deepfake detection and SASV under diverse conditions. It defines two tracks (Track 1: standalone CM; Track 2: SASV) and closed/open training conditions, with metrics including $minDCF$, $actDCF$, $C_{llr}$, $min a-DCF$, $t$-DCF, and $t$-EER. Key findings show strong gains using foundation models in open settings, but calibration issues persist and cross-dataset generalization remains challenging. The work highlights data-centric design, calibration-aware evaluation, and the need for broader data diversity and architectural diversification for robust detection.

Abstract

ASVspoof 5 is the fifth edition in a series of challenges which promote the study of speech spoofing and deepfake detection solutions. A significant change from previous challenge editions is a new crowdsourced database collected from a substantially greater number of speakers under diverse recording conditions, and a mix of cutting-edge and legacy generative speech technology. With the new database described elsewhere, we provide in this paper an overview of the ASVspoof 5 challenge results for the submissions of 53 participating teams. While many solutions perform well, performance degrades under adversarial attacks and the application of neural encoding/compression schemes. Together with a review of post-challenge results, we also report a study of calibration in addition to other principal challenges and outline a road-map for the future of ASVspoof.

Figures (9)

• Figure 1: Results of ASVspoof 5 challenge Track 1. Ensemble and single systems are marked by $\bullet$ and $\circ$, respectively. Open-condition submissions using and not using pre-trained foundation models are marked by $\blacktriangle$ and $\triangle$, respectively. Note that a system's actDCF value is no smaller than its minDCF value.
• Figure 2: Boxplots of evaluation set minDCF of Track 1. In sub-figure (a), each box shows the raw minDCF values of top 50% submissions in the closed condition. Markers are top-1 submission ($\diamond$), top-2 (o), and top-3 ($\triangleleft$) submissions. The annotated arrows '+ M.f.' and '+ M.c.' mean that attacks are the right hand side are obtained via applying Malafide annd Malacopula, respectively, to the attacks on the left hand side. Figures for other tracks and conditions are presented in the appendix. In sub-figure (b), the median minDCF value of the top 50% submissions for each attack is computed, and each box summarizes the median minDCF values of the attacks in the group (either TTS, VC, or adversarial). Markers are easiest ($\blacklozenge$) and most hardest ($\bullet$) attacks. In sub-figure (c), each box shows the raw minDCF values of top 50% submissions in a codec condition. Markers are the same as (a).
• Figure 3: Results of ASVspoof 5 challenge Track 2. Ensemble systems and single systems are marked by $\bullet$ and $\circ$, respectively. Open-condition submissions using and not using pre-trained self-supervised models are marked by $\blacktriangle$ and $\triangle$, respectively. System REF refers to the organisers’ ASV without a CM. Results of t-DCF and t-EER are presented if the system submitted the optional CM and ASV scores.
• Figure 4: Values of normalized DCF at different decision thresholds (§ \ref{['sec:dis:calibration_cm']}). The blue vertical line marks the threshold for Track 1 actDCF computation. The shaded area is upper-bounded by the normalized DCF of a dummy CM that rejects or accept all trials.
• Figure 5: Distributions of CM scores from submission T45 (left) and T24 (right) in Track 1 open condition. The blue and black vertical lines correspond to the Bayesian decision threshold and the one achieving the min DCF, respectively.