Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition

Yi Chang, Zhao Ren, Zixing Zhang, Xin Jing, Kun Qian, Xi Shao, Bin Hu, Tanja Schultz, Björn W. Schuller

TL;DR

This work addresses the vulnerability of speech emotion recognition (SER) models to adversarial perturbations by introducing STAA-Net, a generator-based attacker that creates sparse, transferable perturbations in an end-to-end fashion. The method employs a Wave-U-Net–style generator to produce perturbation magnitudes $\mathbf{v}$ and positions $\mathbf{m}$, forming $\boldsymbol{\delta}=\mathbf{v}\otimes\mathbf{m}$ under an $\ell_{\infty}$ bound $\epsilon$ and sparsity via $\boldsymbol{\delta}$ factorisation, trained with a combination of $\mathcal{L}_{adv}$ (Carlini–Wagner), $\mathcal{L}_{mag}$, $\mathcal{L}_{spa}$, and $\mathcal{L}_{qua}$. Evaluations on DEMoS and IEMOCAP show that STAA-Net delivers fast perturbation generation (≈0.01 s), high attack success rates with relatively sparse perturbations (low $\ell_0$-like impact), and notable cross-model transferability, outperforming several baselines in sparsity and efficiency while maintaining imperceptibility (high $SNR$). These findings highlight practical security concerns for SER systems and motivate robust defense strategies, with future work proposed on targeted attacks, broader audio tasks, defense mechanisms, and automatic loss-weight tuning.

Abstract

Speech contains rich information on the emotions of humans, and Speech Emotion Recognition (SER) has been an important topic in the area of human-computer interaction. The robustness of SER models is crucial, particularly in privacy-sensitive and reliability-demanding domains like private healthcare. Recently, the vulnerability of deep neural networks in the audio domain to adversarial attacks has become a popular area of research. However, prior works on adversarial attacks in the audio domain primarily rely on iterative gradient-based techniques, which are time-consuming and prone to overfitting the specific threat model. Furthermore, the exploration of sparse perturbations, which have the potential for better stealthiness, remains limited in the audio domain. To address these challenges, we propose a generator-based attack method to generate sparse and transferable adversarial examples to deceive SER models in an end-to-end and efficient manner. We evaluate our method on two widely-used SER datasets, Database of Elicited Mood in Speech (DEMoS) and Interactive Emotional dyadic MOtion CAPture (IEMOCAP), and demonstrate its ability to generate successful sparse adversarial examples in an efficient manner. Moreover, our generated adversarial examples exhibit model-agnostic transferability, enabling effective adversarial attacks on advanced victim models.

STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition

TL;DR

This work addresses the vulnerability of speech emotion recognition (SER) models to adversarial perturbations by introducing STAA-Net, a generator-based attacker that creates sparse, transferable perturbations in an end-to-end fashion. The method employs a Wave-U-Net–style generator to produce perturbation magnitudes and positions , forming under an bound and sparsity via factorisation, trained with a combination of (Carlini–Wagner), , , and . Evaluations on DEMoS and IEMOCAP show that STAA-Net delivers fast perturbation generation (≈0.01 s), high attack success rates with relatively sparse perturbations (low -like impact), and notable cross-model transferability, outperforming several baselines in sparsity and efficiency while maintaining imperceptibility (high ). These findings highlight practical security concerns for SER systems and motivate robust defense strategies, with future work proposed on targeted attacks, broader audio tasks, defense mechanisms, and automatic loss-weight tuning.

Abstract

Speech contains rich information on the emotions of humans, and Speech Emotion Recognition (SER) has been an important topic in the area of human-computer interaction. The robustness of SER models is crucial, particularly in privacy-sensitive and reliability-demanding domains like private healthcare. Recently, the vulnerability of deep neural networks in the audio domain to adversarial attacks has become a popular area of research. However, prior works on adversarial attacks in the audio domain primarily rely on iterative gradient-based techniques, which are time-consuming and prone to overfitting the specific threat model. Furthermore, the exploration of sparse perturbations, which have the potential for better stealthiness, remains limited in the audio domain. To address these challenges, we propose a generator-based attack method to generate sparse and transferable adversarial examples to deceive SER models in an end-to-end and efficient manner. We evaluate our method on two widely-used SER datasets, Database of Elicited Mood in Speech (DEMoS) and Interactive Emotional dyadic MOtion CAPture (IEMOCAP), and demonstrate its ability to generate successful sparse adversarial examples in an efficient manner. Moreover, our generated adversarial examples exhibit model-agnostic transferability, enabling effective adversarial attacks on advanced victim models.
Paper Structure (22 sections, 9 equations, 3 figures, 6 tables)

This paper contains 22 sections, 9 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. End-to-end SER
  4. Adversarial Attacks in the Audio Domain
  5. Methods
  6. Problem Formulation
  7. Proposed Approach
  8. Overall Architecture
  9. Wave-U-Net
  10. Perturbation Magnitudes and Positions
  11. Training of the Generator
  12. Loss Functions Design
  13. Experimental Implementations
  14. Datasets
  15. Applied Adversarial Attack Baselines
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Overall architecture of the proposed STAA-Net.
  • Figure 2: Wave-U-Net architecture. $\rightarrow$ describes down- and up-sampling; $\dashrightarrow$ means concatenation.
  • Figure 3: Comparison of the waveforms and log Mel spectrograms of one original audio sample from the DEMoS dataset and its STAA-Net generated adversarial example. The file name of the sample is 'NP_f_43_pau05b.wav', spoken by a female with the emotion class 'fear'.