Double Multi-Head Attention Multimodal System for Odyssey 2024 Speech Emotion Recognition Challenge

TL;DR

This work tackles robust Speech Emotion Recognition (SER) in realistic settings by introducing a Double Multi-Head Attention Multimodal System that fuses self-supervised acoustic representations with text information derived from Whisper transcripts. The architecture employs a first MHA layer (standard or sub-vector) to learn contextualized multimodal features, followed by a second attention pooling to produce an utterance-level vector that is classified into eight emotions; online data augmentation and a three-model hard ensemble further enhance performance. A key contribution is the comparison between standard and sub-vector MHA within a multimodal fusion framework, highlighting the parameter efficiency of the sub-vector variant while maintaining competitive accuracy. The method achieves third place in the Odyssey 2024 Task 1 challenge, outperforming the official baseline and demonstrating the effectiveness of DMHA-based multimodal fusion for SER, especially under data-constrained conditions.

Abstract

As computer-based applications are becoming more integrated into our daily lives, the importance of Speech Emotion Recognition (SER) has increased significantly. Promoting research with innovative approaches in SER, the Odyssey 2024 Speech Emotion Recognition Challenge was organized as part of the Odyssey 2024 Speaker and Language Recognition Workshop. In this paper we describe the Double Multi-Head Attention Multimodal System developed for this challenge. Pre-trained self-supervised models were used to extract informative acoustic and text features. An early fusion strategy was adopted, where a Multi-Head Attention layer transforms these mixed features into complementary contextualized representations. A second attention mechanism is then applied to pool these representations into an utterance-level vector. Our proposed system achieved the third position in the categorical task ranking with a 34.41% Macro-F1 score, where 31 teams participated in total.

Figures (4)

• Figure 1: Double Multi-Head Attention Multimodal System architecture.
• Figure 2: Validation confusion matrix of the ensemble model.
• Figure 3: Head 0 attention weights visualization. The white vertical dashed line divides speech features from text features.
• Figure 4: Head 1 attention weights visualization. The white vertical dashed line divides speech features from text features.