Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions

Alex-Răzvan Ispas, Théo Deschamps-Berger, Laurence Devillers

TL;DR

This work tackles speech emotion recognition by jointly predicting categorical labels and dimensional valence/arousal in spontaneous conversations. It introduces a multi-task, multi-modal architecture that fuses acoustic (HuBERT-large) and linguistic (DeBERTaV3-large) features via two cross-attention layers and learnable bridge tokens, with a Random Modality Masking strategy to balance uni- and multi-modal learning. The multi-task objective combines categorical cross-entropy with CCC-based losses for valence and arousal, leading to improved valence performance (CCC up to $0.748$) and competitive categorical accuracy (WAR ~74%). The findings demonstrate that cross-regularisation between tasks and bridge-token-based fusion significantly benefits emotion estimation, offering a robust approach for SER in naturalistic data and establishing a path for further refinements in token design and regression strategies.

Abstract

Speech emotion recognition (SER) has received a great deal of attention in recent years in the context of spontaneous conversations. While there have been notable results on datasets like the well known corpus of naturalistic dyadic conversations, IEMOCAP, for both the case of categorical and dimensional emotions, there are few papers which try to predict both paradigms at the same time. Therefore, in this work, we aim to highlight the performance contribution of multi-task learning by proposing a multi-task, multi-modal system that predicts categorical and dimensional emotions. The results emphasise the importance of cross-regularisation between the two types of emotions. Our approach consists of a multi-task, multi-modal architecture that uses parallel feature refinement through self-attention for the feature of each modality. In order to fuse the features, our model introduces a set of learnable bridge tokens that merge the acoustic and linguistic features with the help of cross-attention. Our experiments for categorical emotions on 10-fold validation yield results comparable to the current state-of-the-art. In our configuration, our multi-task approach provides better results compared to learning each paradigm separately. On top of that, our best performing model achieves a high result for valence compared to the previous multi-task experiments.

A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions

TL;DR

This work tackles speech emotion recognition by jointly predicting categorical labels and dimensional valence/arousal in spontaneous conversations. It introduces a multi-task, multi-modal architecture that fuses acoustic (HuBERT-large) and linguistic (DeBERTaV3-large) features via two cross-attention layers and learnable bridge tokens, with a Random Modality Masking strategy to balance uni- and multi-modal learning. The multi-task objective combines categorical cross-entropy with CCC-based losses for valence and arousal, leading to improved valence performance (CCC up to ) and competitive categorical accuracy (WAR ~74%). The findings demonstrate that cross-regularisation between tasks and bridge-token-based fusion significantly benefits emotion estimation, offering a robust approach for SER in naturalistic data and establishing a path for further refinements in token design and regression strategies.

Abstract

Speech emotion recognition (SER) has received a great deal of attention in recent years in the context of spontaneous conversations. While there have been notable results on datasets like the well known corpus of naturalistic dyadic conversations, IEMOCAP, for both the case of categorical and dimensional emotions, there are few papers which try to predict both paradigms at the same time. Therefore, in this work, we aim to highlight the performance contribution of multi-task learning by proposing a multi-task, multi-modal system that predicts categorical and dimensional emotions. The results emphasise the importance of cross-regularisation between the two types of emotions. Our approach consists of a multi-task, multi-modal architecture that uses parallel feature refinement through self-attention for the feature of each modality. In order to fuse the features, our model introduces a set of learnable bridge tokens that merge the acoustic and linguistic features with the help of cross-attention. Our experiments for categorical emotions on 10-fold validation yield results comparable to the current state-of-the-art. In our configuration, our multi-task approach provides better results compared to learning each paradigm separately. On top of that, our best performing model achieves a high result for valence compared to the previous multi-task experiments.
Paper Structure (16 sections, 6 equations, 2 figures, 3 tables)

This paper contains 16 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Multi-task Learning
  5. Uni-modal Refinement
  6. Modality Interaction
  7. Fusing the results of the multi-head cross-attention layers
  8. Random Modality Masking
  9. Experiment Design
  10. Dataset
  11. Experimental Set Up
  12. Results
  13. Comparing Results
  14. Our Results
  15. Conclusion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: The head consists of a classifier and a regressor. The regressor has two sigmoid neurons which predict the valence and the arousal.
  • Figure 2: Masking chance cosine decay alongside the training of a fold. When the masking chance gets below 0.1, it is set by default to 0