Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Emotion-Invariant Speaker Representations for Speaker Verification

Jingguang Tian, Xinhui Hu, Xinkang Xu

TL;DR

This work tackles the vulnerability of speaker verification to emotional variation by training emotion-invariant speaker representations. It introduces CopyPaste-based parallel data augmentation to create emotional diversity, an Emotion-invariant Representation Learning (ERL) objective that blends AAM-Softmax with a cosine-similarity penalty to reduce emotion-related variance, and an emotion-aware masking strategy that targets emotionally salient regions via RMS energy. The method is trained on VoxCeleb for pretraining and Dusha for emotion-rich finetuning, with extensive ablations showing the additive gains from CopyPaste, ERL, and EM, culminating in a relative EER reduction of 19.29% on the merged emotional test set. The results suggest strong potential for robust speaker verification in real-world, emotionally varied speech scenarios.

Abstract

In recent years, the rapid progress in speaker verification (SV) technology has been driven by the extraction of speaker representations based on deep learning. However, such representations are still vulnerable to emotion variability. To address this issue, we propose multiple improvements to train speaker encoders to increase emotion robustness. Firstly, we utilize CopyPaste-based data augmentation to gather additional parallel data, which includes different emotional expressions from the same speaker. Secondly, we apply cosine similarity loss to restrict parallel sample pairs and minimize intra-class variation of speaker representations to reduce their correlation with emotional information. Finally, we use emotion-aware masking (EM) based on the speech signal energy on the input parallel samples to further strengthen the speaker representation and make it emotion-invariant. We conduct a comprehensive ablation study to demonstrate the effectiveness of these various components. Experimental results show that our proposed method achieves a relative 19.29\% drop in EER compared to the baseline system.

Learning Emotion-Invariant Speaker Representations for Speaker Verification

TL;DR

This work tackles the vulnerability of speaker verification to emotional variation by training emotion-invariant speaker representations. It introduces CopyPaste-based parallel data augmentation to create emotional diversity, an Emotion-invariant Representation Learning (ERL) objective that blends AAM-Softmax with a cosine-similarity penalty to reduce emotion-related variance, and an emotion-aware masking strategy that targets emotionally salient regions via RMS energy. The method is trained on VoxCeleb for pretraining and Dusha for emotion-rich finetuning, with extensive ablations showing the additive gains from CopyPaste, ERL, and EM, culminating in a relative EER reduction of 19.29% on the merged emotional test set. The results suggest strong potential for robust speaker verification in real-world, emotionally varied speech scenarios.

Abstract

In recent years, the rapid progress in speaker verification (SV) technology has been driven by the extraction of speaker representations based on deep learning. However, such representations are still vulnerable to emotion variability. To address this issue, we propose multiple improvements to train speaker encoders to increase emotion robustness. Firstly, we utilize CopyPaste-based data augmentation to gather additional parallel data, which includes different emotional expressions from the same speaker. Secondly, we apply cosine similarity loss to restrict parallel sample pairs and minimize intra-class variation of speaker representations to reduce their correlation with emotional information. Finally, we use emotion-aware masking (EM) based on the speech signal energy on the input parallel samples to further strengthen the speaker representation and make it emotion-invariant. We conduct a comprehensive ablation study to demonstrate the effectiveness of these various components. Experimental results show that our proposed method achieves a relative 19.29\% drop in EER compared to the baseline system.

Paper Structure

This paper contains 17 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. LEARNING EMOTION-INVARIANT SPEAKER REPRESENTATIONS
  3. CopyPaste-based Data Augmentation
  4. Emotion-invariant Representation Learning
  5. Emotion-aware Masking
  6. EXPERIMENTAL SETUP
  7. Datasets
  8. Baseline System
  9. Training Details and Evaluation Metric
  10. RESULTS
  11. Performance of the baseline system
  12. Performance of the proposed system
  13. Ablation studies
  14. Comparison of different data augmentation methods
  15. Effectiveness of the ERL
  16. ...and 2 more sections

Figures (1)

  • Figure 1: The proposed scheme for learning emotion-invariant speaker representations. Pairs of training samples are fed into a Siamese network with the shared parameter set. ${x}'$ is the result of applying CopyPaste to $x$.