Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Grained Frame Modeling in Multi-head Self-Attention for Speech Deepfake Detection

Tuan Dat Phuong, Duc-Tuan Truong, Long-Vu Hoang, Trang Nguyen Thi Thu

TL;DR

This paper tackles the challenge of detecting synthetic speech artifacts that are localized in time by enhancing MHSA-based detectors with Fine-Grained Frame Modeling (FGFM). FGFM introduces Multi-Head Voting (MHV) to select informative frames per attention head and Cross-Layer Refinement (CLR) to aggregate discriminative cues across layers, with a Dynamic Aggregation Feed-Forward (DAFF) module to enrich the classification token. Empirical results on ASVspoof 2019/2021 LA/DF and ITW show state-of-the-art EERs, including $0.90\%$, $1.88\%$, and $6.64\%$ on $21LA$, $21DF$, and ITW respectively, along with substantial relative improvements over baselines and strong generalization to Transformer backbones. The findings demonstrate that selective information modeling and cross-layer fusion substantially improve robustness to diverse spoofing conditions in speech deepfake detection.

Abstract

Transformer-based models have shown strong performance in speech deepfake detection, largely due to the effectiveness of the multi-head self-attention (MHSA) mechanism. MHSA provides frame-level attention scores, which are particularly valuable because deepfake artifacts often occur in small, localized regions along the temporal dimension of speech. This makes fine-grained frame modeling essential for accurately detecting subtle spoofing cues. In this work, we propose fine-grained frame modeling (FGFM) for MHSA-based speech deepfake detection, where the most informative frames are first selected through a multi-head voting (MHV) module. These selected frames are then refined via a cross-layer refinement (CLR) module to enhance the model's ability to learn subtle spoofing cues. Experimental results demonstrate that our method outperforms the baseline model and achieves Equal Error Rate (EER) of 0.90%, 1.88%, and 6.64% on the LA21, DF21, and ITW datasets, respectively. These consistent improvements across multiple benchmarks highlight the effectiveness of our fine-grained modeling for robust speech deepfake detection.

Fine-Grained Frame Modeling in Multi-head Self-Attention for Speech Deepfake Detection

TL;DR

This paper tackles the challenge of detecting synthetic speech artifacts that are localized in time by enhancing MHSA-based detectors with Fine-Grained Frame Modeling (FGFM). FGFM introduces Multi-Head Voting (MHV) to select informative frames per attention head and Cross-Layer Refinement (CLR) to aggregate discriminative cues across layers, with a Dynamic Aggregation Feed-Forward (DAFF) module to enrich the classification token. Empirical results on ASVspoof 2019/2021 LA/DF and ITW show state-of-the-art EERs, including , , and on , , and ITW respectively, along with substantial relative improvements over baselines and strong generalization to Transformer backbones. The findings demonstrate that selective information modeling and cross-layer fusion substantially improve robustness to diverse spoofing conditions in speech deepfake detection.

Abstract

Transformer-based models have shown strong performance in speech deepfake detection, largely due to the effectiveness of the multi-head self-attention (MHSA) mechanism. MHSA provides frame-level attention scores, which are particularly valuable because deepfake artifacts often occur in small, localized regions along the temporal dimension of speech. This makes fine-grained frame modeling essential for accurately detecting subtle spoofing cues. In this work, we propose fine-grained frame modeling (FGFM) for MHSA-based speech deepfake detection, where the most informative frames are first selected through a multi-head voting (MHV) module. These selected frames are then refined via a cross-layer refinement (CLR) module to enhance the model's ability to learn subtle spoofing cues. Experimental results demonstrate that our method outperforms the baseline model and achieves Equal Error Rate (EER) of 0.90%, 1.88%, and 6.64% on the LA21, DF21, and ITW datasets, respectively. These consistent improvements across multiple benchmarks highlight the effectiveness of our fine-grained modeling for robust speech deepfake detection.
Paper Structure (15 sections, 2 equations, 2 figures, 3 tables)

This paper contains 15 sections, 2 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. The baseline model
  4. The proposed model
  5. Multi-head voting (MHV) module
  6. Cross-layer refinement (CLR) module.
  7. Experiments
  8. Datasets and experimental setup
  9. Comparison with SOTA models
  10. Ablation studies
  11. The effectiveness of each proposed module
  12. The number of voted frames per each head in MHV module
  13. Key features that models focus on in speech
  14. Conclusion
  15. Acknowledgement

Figures (2)

  • Figure 1: Overview of the proposed Fine-Grained Frame Modeling (FGFM) architecture.
  • Figure 2: Visualization of selected frames (red vertical lines) in MHV, in relation to spectrograms for bonafide (top) and deepfake (bottom) audio samples.