Accuracy enhancement method for speech emotion recognition from spectrogram using temporal frequency correlation and positional information learning through knowledge transfer

TL;DR

A method to improve the accuracy of speech emotion recognition (SER) by using vision transformer (ViT) to attend to the correlation of frequency with time (x-axis) in spectrogram and transferring positional information between ViT through knowledge transfer is proposed.

Abstract

In this paper, we propose a method to improve the accuracy of speech emotion recognition (SER) by using vision transformer (ViT) to attend to the correlation of frequency (y-axis) with time (x-axis) in spectrogram and transferring positional information between ViT through knowledge transfer. The proposed method has the following originality i) We use vertically segmented patches of log-Mel spectrogram to analyze the correlation of frequencies over time. This type of patch allows us to correlate the most relevant frequencies for a particular emotion with the time they were uttered. ii) We propose the use of image coordinate encoding, an absolute positional encoding suitable for ViT. By normalizing the x, y coordinates of the image to -1 to 1 and concatenating them to the image, we can effectively provide valid absolute positional information for ViT. iii) Through feature map matching, the locality and location information of the teacher network is effectively transmitted to the student network. Teacher network is a ViT that contains locality of convolutional stem and absolute position information through image coordinate encoding, and student network is a structure that lacks positional encoding in the basic ViT structure. In feature map matching stage, we train through the mean absolute error (L1 loss) to minimize the difference between the feature maps of the two networks. To validate the proposed method, three emotion datasets (SAVEE, EmoDB, and CREMA-D) consisting of speech were converted into log-Mel spectrograms for comparison experiments. The experimental results show that the proposed method significantly outperforms the state-of-the-art methods in terms of weighted accuracy while requiring significantly fewer floating point operations (FLOPs). Overall, the proposed method offers an promising solution for SER by providing improved efficiency and performance.

Figures (6)

• Figure 1: The overall process of proposed method in this paper. i: (a) Train the teacher network using Cross entropy loss. ii: (b) Match the feature maps of the student network and the trained teacher network using Mean absolute error(L1 loss). iii: (c) Train the student network with the feature map matching performed using cross entropy loss + $\alpha$ * L1 loss. Each stage is performed repeatedly with 50 epochs.
• Figure 2: Structure of convolutional stem in this paper. The numbers above each block represent the output channels. To ensure that no information is lost, the after convolution is equal to the input size of 128 x 128.
• Figure 3: Image coordinate encoding. The x, y coordinates of the image are normalized to the range -1 to 1, (a) and (b) concatenated to a log-Mel spectrogram after convolutional stem, (c).
• Figure 4: The structure of the teacher network. It contains a convolutional stem to obtain locality and an image coordinate encoding to disambiguate the location information.
• Figure 5: Structure of the student network. To verify that locality inference and positional encoding is possible through transfer learning without convolutional stem and image coordinate encoding, we construct a basic ViT.