Bridging Discrete and Continuous: A Multimodal Strategy for Complex Emotion Detection

TL;DR

The paper tackles the challenge of capturing rich emotional states by bridging discrete and continuous representations using a multimodal framework that leverages facial, vocal, and textual cues. By embedding emotions in a three-dimensional Valence-Arousal-Dominance space and transforming discrete labels via a K-means classifier, the approach supports both closed-set and open-set emotion recognition and enables nuanced open-vocabulary generation. Evaluated on the MER2024 dataset, the proposed VAD-based model achieves competitive continuous predictions (L2 $=0.64$, MSE $=0.19$, MAE $=0.36$, PCC $=0.47$) and improves discrete classification precision/recall relative to a baseline, while also demonstrating semantically meaningful open-vocabulary outputs. The work advances multimodal emotion understanding with a practical pipeline for translating between discrete and continuous emotion vocabularies, and highlights avenues for broader cultural applicability and richer temporal dynamics in future research.

Abstract

In the domain of human-computer interaction, accurately recognizing and interpreting human emotions is crucial yet challenging due to the complexity and subtlety of emotional expressions. This study explores the potential for detecting a rich and flexible range of emotions through a multimodal approach which integrates facial expressions, voice tones, and transcript from video clips. We propose a novel framework that maps variety of emotions in a three-dimensional Valence-Arousal-Dominance (VAD) space, which could reflect the fluctuations and positivity/negativity of emotions to enable a more variety and comprehensive representation of emotional states. We employed K-means clustering to transit emotions from traditional discrete categorization to a continuous labeling system and built a classifier for emotion recognition upon this system. The effectiveness of the proposed model is evaluated using the MER2024 dataset, which contains culturally consistent video clips from Chinese movies and TV series, annotated with both discrete and open-vocabulary emotion labels. Our experiment successfully achieved the transformation between discrete and continuous models, and the proposed model generated a more diverse and comprehensive set of emotion vocabulary while maintaining strong accuracy.

Figures (2)

• Figure 1: Emotion Vocabularies in 3D VAD Space
• Figure 2: VAD Model Pipline