Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Strong Alone, Stronger Together: Synergizing Modality-Binding Foundation Models with Optimal Transport for Non-Verbal Emotion Recognition

Orchid Chetia Phukan, Mohd Mujtaba Akhtar, Girish, Swarup Ranjan Behera, Sishir Kalita, Arun Balaji Buduru, Rajesh Sharma, S. R Mahadeva Prasanna

TL;DR

This study hypothesizes that MFMs, with their joint pre-training across multiple modalities, will be more effective in non-verbal sounds emotion recognition (NVER) by better interpreting and differentiating subtle emotional cues that may be ambiguous in audio-only foundation models (AFMs).

Abstract

In this study, we investigate multimodal foundation models (MFMs) for emotion recognition from non-verbal sounds. We hypothesize that MFMs, with their joint pre-training across multiple modalities, will be more effective in non-verbal sounds emotion recognition (NVER) by better interpreting and differentiating subtle emotional cues that may be ambiguous in audio-only foundation models (AFMs). To validate our hypothesis, we extract representations from state-of-the-art (SOTA) MFMs and AFMs and evaluated them on benchmark NVER datasets. We also investigate the potential of combining selected foundation model representations to enhance NVER further inspired by research in speech recognition and audio deepfake detection. To achieve this, we propose a framework called MATA (Intra-Modality Alignment through Transport Attention). Through MATA coupled with the combination of MFMs: LanguageBind and ImageBind, we report the topmost performance with accuracies of 76.47%, 77.40%, 75.12% and F1-scores of 70.35%, 76.19%, 74.63% for ASVP-ESD, JNV, and VIVAE datasets against individual FMs and baseline fusion techniques and report SOTA on the benchmark datasets.

Strong Alone, Stronger Together: Synergizing Modality-Binding Foundation Models with Optimal Transport for Non-Verbal Emotion Recognition

TL;DR

This study hypothesizes that MFMs, with their joint pre-training across multiple modalities, will be more effective in non-verbal sounds emotion recognition (NVER) by better interpreting and differentiating subtle emotional cues that may be ambiguous in audio-only foundation models (AFMs).

Abstract

In this study, we investigate multimodal foundation models (MFMs) for emotion recognition from non-verbal sounds. We hypothesize that MFMs, with their joint pre-training across multiple modalities, will be more effective in non-verbal sounds emotion recognition (NVER) by better interpreting and differentiating subtle emotional cues that may be ambiguous in audio-only foundation models (AFMs). To validate our hypothesis, we extract representations from state-of-the-art (SOTA) MFMs and AFMs and evaluated them on benchmark NVER datasets. We also investigate the potential of combining selected foundation model representations to enhance NVER further inspired by research in speech recognition and audio deepfake detection. To achieve this, we propose a framework called MATA (Intra-Modality Alignment through Transport Attention). Through MATA coupled with the combination of MFMs: LanguageBind and ImageBind, we report the topmost performance with accuracies of 76.47%, 77.40%, 75.12% and F1-scores of 70.35%, 76.19%, 74.63% for ASVP-ESD, JNV, and VIVAE datasets against individual FMs and baseline fusion techniques and report SOTA on the benchmark datasets.
Paper Structure (12 sections, 2 equations, 7 figures, 1 table)

This paper contains 12 sections, 2 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Foundation Models
  3. Multimodal Foundation Models
  4. Audio Foundation Models
  5. Modeling
  6. Individual Foundation Models
  7. Modality Alignment through Transport Attention (MATA)
  8. Experiments
  9. Benchmark Datasets
  10. Training Details
  11. Results and Discussion
  12. Conclusion

Figures (7)

  • Figure 1: MATA framework: OT and FCN stand for Optimal Transport and Fully Connected Network, respectively. FM1 and FM2 refer to Foundation Model 1 and 2; U11 and U22 represent features from individual FM branches, while U12 and U21 represent features transported from FM2 to the FM1 network and from FM1 to the FM2 network, respectively.
  • Figure 3: Confusion Matrix of MATA: UNI, WA, LB, and IB stand for Unispeech-SAT, WavLM, LanguageBind, and ImageBind, respectively.
  • Figure : (a) ImageBind
  • Figure : (a) ImageBind
  • Figure : (b) LanguageBind
  • ...and 2 more figures