Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Effective Energy Mask-based Adversarial Evasion Attacks against Misclassification in Speaker Recognition Systems

Chanwoo Park, Chanwoo Kim

TL;DR

This work tackles adversarial evasion in speaker recognition by introducing masked energy perturbation (MEP), which perturbs small energy regions in the mel-filterbank domain under psychoacoustic constraints. A two-part methodology combines SEM energy masking with gradient-based perturbation to produce adversarial examples, including an Iterative variant (I-MEP) that refines perturbations over multiple steps. Evaluations on LibriSpeech with ResNetSE34-L, ResNetSE34-V, and ECAPA-TDNN show that MEP and especially I-MEP achieve strong evasion while preserving audio quality (PESQ $> 3.5$ and SNR around $38$ dB) and increasing equal error rate (EER) in speaker verification. The results highlight a significant gap between perceptual naturalness and vulnerability, underscoring the need for defenses in psychoacoustically informed adversarial settings.

Abstract

Evasion attacks pose significant threats to AI systems, exploiting vulnerabilities in machine learning models to bypass detection mechanisms. The widespread use of voice data, including deepfakes, in promising future industries is currently hindered by insufficient legal frameworks. Adversarial attack methods have emerged as the most effective countermeasure against the indiscriminate use of such data. This research introduces masked energy perturbation (MEP), a novel approach using power spectrum for energy masking of original voice data. MEP applies masking to small energy regions in the frequency domain before generating adversarial perturbations, targeting areas less noticeable to the human auditory model. The study primarily employs advanced speaker recognition models, including ECAPA-TDNN and ResNet34, which have shown remarkable performance in speaker verification tasks. The proposed MEP method demonstrated strong performance in both audio quality and evasion effectiveness. The energy masking approach effectively minimizes the perceptual evaluation of speech quality (PESQ) degradation, indicating that minimal perceptual distortion occurs to the human listener despite the adversarial perturbations. Specifically, in the PESQ evaluation, the relative performance of the MEP method was 26.68% when compared to the fast gradient sign method (FGSM) and iterative FGSM.

An Effective Energy Mask-based Adversarial Evasion Attacks against Misclassification in Speaker Recognition Systems

TL;DR

This work tackles adversarial evasion in speaker recognition by introducing masked energy perturbation (MEP), which perturbs small energy regions in the mel-filterbank domain under psychoacoustic constraints. A two-part methodology combines SEM energy masking with gradient-based perturbation to produce adversarial examples, including an Iterative variant (I-MEP) that refines perturbations over multiple steps. Evaluations on LibriSpeech with ResNetSE34-L, ResNetSE34-V, and ECAPA-TDNN show that MEP and especially I-MEP achieve strong evasion while preserving audio quality (PESQ and SNR around dB) and increasing equal error rate (EER) in speaker verification. The results highlight a significant gap between perceptual naturalness and vulnerability, underscoring the need for defenses in psychoacoustically informed adversarial settings.

Abstract

Evasion attacks pose significant threats to AI systems, exploiting vulnerabilities in machine learning models to bypass detection mechanisms. The widespread use of voice data, including deepfakes, in promising future industries is currently hindered by insufficient legal frameworks. Adversarial attack methods have emerged as the most effective countermeasure against the indiscriminate use of such data. This research introduces masked energy perturbation (MEP), a novel approach using power spectrum for energy masking of original voice data. MEP applies masking to small energy regions in the frequency domain before generating adversarial perturbations, targeting areas less noticeable to the human auditory model. The study primarily employs advanced speaker recognition models, including ECAPA-TDNN and ResNet34, which have shown remarkable performance in speaker verification tasks. The proposed MEP method demonstrated strong performance in both audio quality and evasion effectiveness. The energy masking approach effectively minimizes the perceptual evaluation of speech quality (PESQ) degradation, indicating that minimal perceptual distortion occurs to the human listener despite the adversarial perturbations. Specifically, in the PESQ evaluation, the relative performance of the MEP method was 26.68% when compared to the fast gradient sign method (FGSM) and iterative FGSM.
Paper Structure (11 sections, 7 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 7 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Energy Mask
  4. Evasion Attack
  5. Experiment results and analysis
  6. Experimental Setup
  7. Speaker Encoder Model
  8. Attack Methods
  9. Results of using MEP and attack methods on the speaker encoders
  10. Conclusions
  11. Acknowledgment

Figures (2)

  • Figure 1: Mel spectrograms of a single utterance. The top graph shows the original speech mel spectrogram, and the bottom graph shows the mel spectrogram with the small energy region masked.
  • Figure 2: Block diagram for generating adversarial samples based on gradient descent with masked energy perturbation (MEP). Here, $c$ is the filterbank channel index.