Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TwinShift: Benchmarking Audio Deepfake Detection across Synthesizer and Speaker Shifts

Jiyoung Hong, Yoonseo Chung, Seungyeon Oh, Juntae Kim, Jiyoung Lee, Sookyung Kim, Hyunsoo Cho

TL;DR

Audio deepfake detectors currently fail to generalize to unseen synthesis methods and speakers. TwinShift introduces a dual-axis benchmark with six unseen synthesis systems and disjoint speakers to stress-test robustness, revealing major gaps where generator shifts dominate degradation and cross-environment transfer is highly asymmetric. Across detectors and environments, no single approach delivers universal resilience, and high-fidelity spoofs do not guarantee broad transfer, underscoring the need for holistic improvements in both data diversity and modeling. The benchmark provides a principled framework to guide development of ADD systems capable of withstanding evolving spoofing threats in real-world deployments.

Abstract

Audio deepfakes pose a growing threat, already exploited in fraud and misinformation. A key challenge is ensuring detectors remain robust to unseen synthesis methods and diverse speakers, since generation techniques evolve quickly. Despite strong benchmark results, current systems struggle to generalize to new conditions limiting real-world reliability. To address this, we introduce TWINSHIFT, a benchmark explicitly designed to evaluate detection robustness under strictly unseen conditions. Our benchmark is constructed from six different synthesis systems, each paired with disjoint sets of speakers, allowing for a rigorous assessment of how well detectors generalize when both the generative model and the speaker identity change. Through extensive experiments, we show that TWINSHIFT reveals important robustness gaps, uncover overlooked limitations, and provide principled guidance for developing ADD systems. The TWINSHIFT benchmark can be accessed at https://github.com/intheMeantime/TWINSHIFT.

TwinShift: Benchmarking Audio Deepfake Detection across Synthesizer and Speaker Shifts

TL;DR

Audio deepfake detectors currently fail to generalize to unseen synthesis methods and speakers. TwinShift introduces a dual-axis benchmark with six unseen synthesis systems and disjoint speakers to stress-test robustness, revealing major gaps where generator shifts dominate degradation and cross-environment transfer is highly asymmetric. Across detectors and environments, no single approach delivers universal resilience, and high-fidelity spoofs do not guarantee broad transfer, underscoring the need for holistic improvements in both data diversity and modeling. The benchmark provides a principled framework to guide development of ADD systems capable of withstanding evolving spoofing threats in real-world deployments.

Abstract

Audio deepfakes pose a growing threat, already exploited in fraud and misinformation. A key challenge is ensuring detectors remain robust to unseen synthesis methods and diverse speakers, since generation techniques evolve quickly. Despite strong benchmark results, current systems struggle to generalize to new conditions limiting real-world reliability. To address this, we introduce TWINSHIFT, a benchmark explicitly designed to evaluate detection robustness under strictly unseen conditions. Our benchmark is constructed from six different synthesis systems, each paired with disjoint sets of speakers, allowing for a rigorous assessment of how well detectors generalize when both the generative model and the speaker identity change. Through extensive experiments, we show that TWINSHIFT reveals important robustness gaps, uncover overlooked limitations, and provide principled guidance for developing ADD systems. The TWINSHIFT benchmark can be accessed at https://github.com/intheMeantime/TWINSHIFT.
Paper Structure (13 sections, 1 figure, 3 tables)

This paper contains 13 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary Study
  3. Experimental Setup
  4. Results
  5. TwinShift Benchmark
  6. Experiments
  7. Detectors and Training Protocol
  8. Main Results
  9. Discussion
  10. Transfer is Non-Commutative
  11. Bonafide is Composite; Spoofs Chase Islands
  12. High-fidelity Spoofs Don’t Always Transfer
  13. Conclusion

Figures (1)

  • Figure 1: t-SNE visualizations of audio data. (a) Spoof audio from TwinShift, showing the distribution of different spoofing methods. (b) Zero-shot TTS-generated speech, illustrating how each synthesizer’s output compares to bonafide speech.