Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Task Transformer for Explainable Speech Deepfake Detection via Formant Modeling

Viola Negroni, Luca Cuccovillo, Paolo Bestagini, Patrick Aichroth, Stefano Tubaro

TL;DR

The paper tackles the need for trustworthy, interpretable speech deepfake detection and introduces SFATNet-4, a lightweight multi-task transformer with separate magnitude and phase encoders and three frame-level decoders that jointly predict $F_0$, $F_1$, $F_2$, voicing, and the real/fake label, using time-axis segmentation for efficiency. It demonstrates that the model achieves competitive detection performance while offering built-in frame-level explanations via a pooling attention mechanism and a voicing mask, with robust results across four datasets and varying codecs. The findings show that integrating interpretability into model design can be achieved without sacrificing accuracy, supporting practical deployment in diverse real-world audio environments. Overall, SFATNet-4 advances interpretable, efficient speech deepfake detection by coupling auxiliary prosodic tasks with attention-based explanations.

Abstract

In this work, we introduce a multi-task transformer for speech deepfake detection, capable of predicting formant trajectories and voicing patterns over time, ultimately classifying speech as real or fake, and highlighting whether its decisions rely more on voiced or unvoiced regions. Building on a prior speaker-formant transformer architecture, we streamline the model with an improved input segmentation strategy, redesign the decoding process, and integrate built-in explainability. Compared to the baseline, our model requires fewer parameters, trains faster, and provides better interpretability, without sacrificing prediction performance.

Multi-Task Transformer for Explainable Speech Deepfake Detection via Formant Modeling

TL;DR

The paper tackles the need for trustworthy, interpretable speech deepfake detection and introduces SFATNet-4, a lightweight multi-task transformer with separate magnitude and phase encoders and three frame-level decoders that jointly predict , , , voicing, and the real/fake label, using time-axis segmentation for efficiency. It demonstrates that the model achieves competitive detection performance while offering built-in frame-level explanations via a pooling attention mechanism and a voicing mask, with robust results across four datasets and varying codecs. The findings show that integrating interpretability into model design can be achieved without sacrificing accuracy, supporting practical deployment in diverse real-world audio environments. Overall, SFATNet-4 advances interpretable, efficient speech deepfake detection by coupling auxiliary prosodic tasks with attention-based explanations.

Abstract

In this work, we introduce a multi-task transformer for speech deepfake detection, capable of predicting formant trajectories and voicing patterns over time, ultimately classifying speech as real or fake, and highlighting whether its decisions rely more on voiced or unvoiced regions. Building on a prior speaker-formant transformer architecture, we streamline the model with an improved input segmentation strategy, redesign the decoding process, and integrate built-in explainability. Compared to the baseline, our model requires fewer parameters, trains faster, and provides better interpretability, without sacrificing prediction performance.
Paper Structure (13 sections, 5 equations, 3 figures, 2 tables)

This paper contains 13 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed system
  3. Problem Formulation
  4. Proposed Architecture
  5. Experimental setup
  6. Datasets
  7. Training Specifications
  8. Results
  9. Performance and generalization
  10. Robustness by design
  11. Explainability analysis
  12. Conclusions
  13. References

Figures (3)

  • Figure 1: Illustrative example of the proposed multi-task transformer speech deepfake detector (SFATNet-4).
  • Figure 2: Example model output for an input spectrogram. Top: original spectrogram with predicted $F_0$, $F_1$, and $F_2$. Bottom: predicted binary voiced/unvoiced segments (voiced: white, unvoiced: black).
  • Figure 3: Relative importance of voiced and unvoiced frames for correctly classified speech. Each pair of bars shows the contributions of voiced and unvoiced frames to the model’s decision in a specific dataset, as weighted by the model’s attention scores. Top: real speech; bottom: fake.