Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

learning discriminative features from spectrograms using center loss for speech emotion recognition

Dongyang Dai, Zhiyong Wu, Runnan Li, Xixin Wu, Jia Jia, Helen Meng

TL;DR

The paper tackles speech emotion recognition (SER) by jointly training with softmax cross-entropy $L_s$ and center loss $L_c$ to learn discriminative features from variable-length spectrograms. It introduces an end-to-end CNN+Bi-RNN architecture that produces a $d$-dimensional feature $z$, optimized with a weighted joint loss $L = L_s + \lambda L_c$ to balance inter- and intra-class separation. Empirical results on IEMOCAP show that center loss yields notable improvements: over 3% in UA and WA on Mel-spectrograms and over 4% on STFT spectrograms, with PCA visualizations confirming tighter same-class clustering. This approach offers a practical, integrable enhancement for SER without requiring pair/triplet sampling, potentially benefiting real-time and multimodal emotion systems.

Abstract

Identifying the emotional state from speech is essential for the natural interaction of the machine with the speaker. However, extracting effective features for emotion recognition is difficult, as emotions are ambiguous. We propose a novel approach to learn discriminative features from variable length spectrograms for emotion recognition by cooperating softmax cross-entropy loss and center loss together. The softmax cross-entropy loss enables features from different emotion categories separable, and center loss efficiently pulls the features belonging to the same emotion category to their center. By combining the two losses together, the discriminative power will be highly enhanced, which leads to network learning more effective features for emotion recognition. As demonstrated by the experimental results, after introducing center loss, both the unweighted accuracy and weighted accuracy are improved by over 3\% on Mel-spectrogram input, and more than 4\% on Short Time Fourier Transform spectrogram input.

learning discriminative features from spectrograms using center loss for speech emotion recognition

TL;DR

The paper tackles speech emotion recognition (SER) by jointly training with softmax cross-entropy and center loss to learn discriminative features from variable-length spectrograms. It introduces an end-to-end CNN+Bi-RNN architecture that produces a -dimensional feature , optimized with a weighted joint loss to balance inter- and intra-class separation. Empirical results on IEMOCAP show that center loss yields notable improvements: over 3% in UA and WA on Mel-spectrograms and over 4% on STFT spectrograms, with PCA visualizations confirming tighter same-class clustering. This approach offers a practical, integrable enhancement for SER without requiring pair/triplet sampling, potentially benefiting real-time and multimodal emotion systems.

Abstract

Identifying the emotional state from speech is essential for the natural interaction of the machine with the speaker. However, extracting effective features for emotion recognition is difficult, as emotions are ambiguous. We propose a novel approach to learn discriminative features from variable length spectrograms for emotion recognition by cooperating softmax cross-entropy loss and center loss together. The softmax cross-entropy loss enables features from different emotion categories separable, and center loss efficiently pulls the features belonging to the same emotion category to their center. By combining the two losses together, the discriminative power will be highly enhanced, which leads to network learning more effective features for emotion recognition. As demonstrated by the experimental results, after introducing center loss, both the unweighted accuracy and weighted accuracy are improved by over 3\% on Mel-spectrogram input, and more than 4\% on Short Time Fourier Transform spectrogram input.
Paper Structure (11 sections, 7 equations, 5 figures, 2 tables)

This paper contains 11 sections, 7 equations, 5 figures, 2 tables.

Table of Contents

  1. introduction
  2. the proposed approach
  3. Model details
  4. Softmax cross-entropy loss
  5. Center loss
  6. Weighted loss and joint loss
  7. experiments and analysis
  8. Experimental setup
  9. Experiments on Mel-Spectrogram
  10. Experiments on STFT spectrogram
  11. conclusions

Figures (5)

  • Figure 1: Model framework. The model takes variable length spectrograms as input and learns discriminative features for SER.
  • Figure 2: Model Details.(a) Model details of CNN layers, (b) Bi-RNN compresses variable length sequence to a fixed-length vector
  • Figure 3: UA and WA on log scale Mel-spectrogram input.(a) model with different $\alpha$ when fixing $\lambda$ = 0.3 (b) model with different $\lambda$ when fixing $\alpha$ = 0.5
  • Figure 4: PCA embedding of features from, (a) training set on setting 1, (b) training set on setting 2, (c) test set on setting 1, (d) test set on setting 2
  • Figure 5: The UA and WA on setting 1 $\sim$ setting 4.