Beyond Identity: A Generalizable Approach for Deepfake Audio Detection

TL;DR

This work tackles the poor cross-dataset generalization of audio deepfake detectors caused by implicit identity leakage. It introduces an identity-independent framework using Artifact Detection Modules (ADMs) and artifact-centric data augmentations, including fixed/dynamic frequency swaps, time-domain alterations, and background noise injection, to emphasize forgery artifacts over speaker traits. Empirical results across ASVspoof2019 LA, ADD 2022, FoR, and In-The-Wild show that ADM-enhanced models consistently improve cross-dataset performance, with Dynamic Frequency Swap often delivering the strongest gains and reducing identity leakage as shown by embedding visualizations. The proposed artifact-based learning approach provides a robust pathway toward more generalizable and deployable audio deepfake detection systems in real-world scenarios.

Abstract

Deepfake audio presents a growing threat to digital security, due to its potential for social engineering, fraud, and identity misuse. However, existing detection models suffer from poor generalization across datasets, due to implicit identity leakage, where models inadvertently learn speaker-specific features instead of manipulation artifacts. To the best of our knowledge, this is the first study to explicitly analyze and address identity leakage in the audio deepfake detection domain. This work proposes an identity-independent audio deepfake detection framework that mitigates identity leakage by encouraging the model to focus on forgery-specific artifacts instead of overfitting to speaker traits. Our approach leverages Artifact Detection Modules (ADMs) to isolate synthetic artifacts in both time and frequency domains, enhancing cross-dataset generalization. We introduce novel dynamic artifact generation techniques, including frequency domain swaps, time domain manipulations, and background noise augmentation, to enforce learning of dataset-invariant features. Extensive experiments conducted on ASVspoof2019, ADD 2022, FoR, and In-The-Wild datasets demonstrate that the proposed ADM-enhanced models achieve F1 scores of 0.230 (ADD 2022), 0.604 (FoR), and 0.813 (In-The-Wild), consistently outperforming the baseline. Dynamic Frequency Swap proves to be the most effective strategy across diverse conditions. These findings emphasize the value of artifact-based learning in mitigating implicit identity leakage for more generalizable audio deepfake detection.

Figures (7)

• Figure 1: Overview of the proposed pipeline. The process begins with standardized audio input, which is converted into Mel-spectrogram images for training an Xception-based baseline model. This baseline is evaluated on multiple datasets to assess its generalization ability. In the second stage, artifacts are introduced into fake audio samples using frequency-based swaps (horizontal and vertical), dynamic frequency swaps, and background noise addition. The Artifact Detection Module (ADM) is trained to differentiate between fake and artifact-injected samples. This module is then fine-tuned for real-fake detection using artifact-informed representations. Finally, cross-dataset evaluations are conducted using the ADM-infused model to measure its effectiveness.
• Figure 2: Mel-spectrogram examples from ASVspoof2019. Dark purple indicates low intensity; bright yellow, high energy.
• Figure 3: Comparison of original and modified fake audio with frequency-domain artifacts. Horizontal distortions result from spectral modifications in the given frequency ranges, exposing inconsistencies in fake speech generation.
• Figure 4: Comparison of original and modified fake audio with time-domain artifacts. Vertical distortions are introduced by replacing audio segments in the given frequency ranges, creating structured disruptions that highlight synthetic inconsistencies.
• Figure 5: Comparison of original and dynamically modified fake samples using Dynamic Frequency Swap. This method injects artifacts by swapping frequency bands at randomized positions per sample, ensuring variation in horizontal artifact bands.