Lightweight Resolution-Aware Audio Deepfake Detection via Cross-Scale Attention and Consistency Learning

TL;DR

This paper addresses robustness gaps in audio deepfake detection under real-world distortions by introducing a resolution-aware framework that explicitly models and aligns multi-resolution spectral representations. It employs cross-scale attention to dynamically fuse resolution-specific embeddings and a consistency loss $\mathcal{L}_{\text{cons}} = \sum_{i < j} \mathbb{E}_{x \sim \mathcal{D}_{\text{real}}} \left[ \lVert \hat{\mathbf{z}}_i - \hat{\mathbf{z}}_j \rVert_2^2 \right]$ to enforce resolution-invariant representations, followed by a linear classifier $\hat{y} = \sigma\left( g\left( \frac{1}{K} \sum_{k=1}^{K} \tilde{\mathbf{z}}_k \right) \right)$. Experiments on ASVspoof 2019 (LA/PA), FoR, and In-the-Wild demonstrate near-ceiling EERs and strong AUC across conditions, with a compact model (~159k parameters and ~936 MFLOPs per forward pass). The results, together with Grad-CAM based interpretability, show that explicit cross-resolution modeling yields robust, scalable, and explainable audio deepfake detection suitable for practical deployment.

Abstract

Audio deepfake detection has become increasingly challenging due to rapid advances in speech synthesis and voice conversion technologies, particularly under channel distortions, replay attacks, and real-world recording conditions. This paper proposes a resolution-aware audio deepfake detection framework that explicitly models and aligns multi-resolution spectral representations through cross-scale attention and consistency learning. Unlike conventional single-resolution or implicit feature-fusion approaches, the proposed method enforces agreement across complementary time--frequency scales. The proposed framework is evaluated on three representative benchmarks: ASVspoof 2019 (LA and PA), the Fake-or-Real (FoR) dataset, and the In-the-Wild Audio Deepfake dataset under a speaker-disjoint protocol. The method achieves near-perfect performance on ASVspoof LA (EER 0.16%), strong robustness on ASVspoof PA (EER 5.09%), FoR rerecorded audio (EER 4.54%), and in-the-wild deepfakes (AUC 0.98, EER 4.81%), significantly outperforming single-resolution and non-attention baselines under challenging conditions. The proposed model remains lightweight and efficient, requiring only 159k parameters and less than 1~GFLOP per inference, making it suitable for practical deployment. Comprehensive ablation studies confirm the critical contributions of cross-scale attention and consistency learning, while gradient-based interpretability analysis reveals that the model learns resolution-consistent and semantically meaningful spectral cues across diverse spoofing conditions. These results demonstrate that explicit cross-resolution modeling provides a principled, robust, and scalable foundation for next-generation audio deepfake detection systems.

Figures (11)

• Figure 1: Multi-resolution log-Mel spectrograms of real and fake audio samples from the In-the-Wild dataset. Low-, mid-, and high-resolution representations reveal complementary temporal and spectral artifacts.
• Figure 2: Overview of the proposed resolution-aware audio deepfake detection framework. An input audio signal is transformed into low-, mid-, and high-resolution Mel spectrograms, which are processed by a shared convolutional encoder to extract resolution-specific embeddings $z_1$, $z_2$, and $z_3$. A cross-scale attention module dynamically fuses these embeddings to form a resolution-invariant representation for classification. During training, a cross-resolution consistency loss is applied to bona fide speech to enforce alignment across resolutions, improving robustness under replay and real-world conditions.
• Figure 3: ASVspoof 2019 LA results: confusion matrix (left), ROC curve (center), and t-SNE embedding of development set representations (right).
• Figure 4: ASVspoof 2019 PA results: confusion matrix (left), ROC curve (center), and t-SNE embedding of development set representations (right).
• Figure 5: FoR for-2sec results: confusion matrix (left), ROC curve (center), and t-SNE embedding (right).