Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emotion and Acoustics Should Agree: Cross-Level Inconsistency Analysis for Audio Deepfake Detection

Jinhua Zhang, Zhenqi Jia, Rui Liu

TL;DR

This work tackles audio deepfake detection by shifting from correlation-based cues to explicit cross-level inconsistency between emotional dynamics and acoustic structure. The proposed EAI-ADD architecture combines an Emotion–Acoustic Alignment Module (EAAM) with an Emotion–Acoustic Inconsistency Modeling Module (EAIMM), leveraging Emotional Variation Amplification Loss (EVAL) and a Hierarchical Inconsistency Graph (HEIG) to capture short-term and cross-scale mismatches. Experiments on ASVspoof 2019LA and 2021LA show state-of-the-art performance, with robust improvements in t-DCF and EER and clear ablation support for each component. The approach highlights the practical value of modeling emotion–acoustic misalignment for spoof detection, though future work should assess real-time and highly compressed scenarios.

Abstract

Audio Deepfake Detection (ADD) aims to detect spoof speech from bonafide speech. Most prior studies assume that stronger correlations within or across acoustic and emotional features imply authenticity, and thus focus on enhancing or measuring such correlations. However, existing methods often treat acoustic and emotional features in isolation or rely on correlation metrics, which overlook subtle desynchronization between them and smooth out abrupt discontinuities. To address these issues, we propose EAI-ADD, which treats cross level emotion acoustic inconsistency as the primary detection signal. We first project emotional and acoustic representations into a comparable space. Then we progressively integrate frame level and utterance level emotion features with acoustic features to capture cross level emotion acoustic inconsistencies across different temporal granularities. Experimental results on the ASVspoof 2019LA and 2021LA datasets demonstrate that the proposed EAI-ADD outperforms baselines, providing a more effective solution for audio anti spoofing detection.

Emotion and Acoustics Should Agree: Cross-Level Inconsistency Analysis for Audio Deepfake Detection

TL;DR

This work tackles audio deepfake detection by shifting from correlation-based cues to explicit cross-level inconsistency between emotional dynamics and acoustic structure. The proposed EAI-ADD architecture combines an Emotion–Acoustic Alignment Module (EAAM) with an Emotion–Acoustic Inconsistency Modeling Module (EAIMM), leveraging Emotional Variation Amplification Loss (EVAL) and a Hierarchical Inconsistency Graph (HEIG) to capture short-term and cross-scale mismatches. Experiments on ASVspoof 2019LA and 2021LA show state-of-the-art performance, with robust improvements in t-DCF and EER and clear ablation support for each component. The approach highlights the practical value of modeling emotion–acoustic misalignment for spoof detection, though future work should assess real-time and highly compressed scenarios.

Abstract

Audio Deepfake Detection (ADD) aims to detect spoof speech from bonafide speech. Most prior studies assume that stronger correlations within or across acoustic and emotional features imply authenticity, and thus focus on enhancing or measuring such correlations. However, existing methods often treat acoustic and emotional features in isolation or rely on correlation metrics, which overlook subtle desynchronization between them and smooth out abrupt discontinuities. To address these issues, we propose EAI-ADD, which treats cross level emotion acoustic inconsistency as the primary detection signal. We first project emotional and acoustic representations into a comparable space. Then we progressively integrate frame level and utterance level emotion features with acoustic features to capture cross level emotion acoustic inconsistencies across different temporal granularities. Experimental results on the ASVspoof 2019LA and 2021LA datasets demonstrate that the proposed EAI-ADD outperforms baselines, providing a more effective solution for audio anti spoofing detection.
Paper Structure (11 sections, 3 equations, 2 figures, 2 tables)

This paper contains 11 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. EAI-ADD Neural Architecture
  3. Feature Extraction
  4. Emotion--Acoustic Alignment Module
  5. Emotion-Acoustic Inconsistency Modeling Module
  6. Overall Objective
  7. Experiments
  8. Experimental Protocol
  9. Main Results
  10. Ablation Study
  11. Conclusions

Figures (2)

  • Figure 1: Frame-level change magnitudes of emotion and acoustic features.
  • Figure 2: Our overall framework is illustrated in (a). (b) illustrates the Emotion-Acoustic Alignment Module. (c) depicts Emotion-Acoustic Inconsistency Modeling Module.