Not all Blends are Equal: The BLEMORE Dataset of Blended Emotion Expressions with Relative Salience Annotations

TL;DR

This paper introduces BlEmoRe, a publicly available multimodal dataset with relative salience annotations for blended emotions, addressing a key gap in existing resources. It comprises 3,050 video clips from 58 actors expressing six basic emotions and all pairwise blends under three salience configurations, enabling two tasks: presence prediction and salience ranking. A broad evaluation of unimodal and multimodal encoders (including VideoMAEv2, HuBERT, WavLM, ImageBind, CLIP, OpenFace, and HiCMAE) shows that multimodal fusion generally yields higher accuracy, with VideoMAEv2 + HuBERT achieving the best presence results ($ACC_{presence} \,\approx\,0.33$) and HiCMAE achieving the best salience results ($ACC_{salience} \,\approx\,0.18$); aggregation-based features outperform subsampling. The work highlights both the potential and challenges of recognizing blended emotions, emphasizing the need for specialized modeling approaches and more diverse data to better capture relative salience in real-world contexts. $ACC_{presence}$ and $ACC_{salience}$ are used as the core evaluation metrics, reflecting presence identification and correct salience ranking respectively.

Abstract

Humans often experience not just a single basic emotion at a time, but rather a blend of several emotions with varying salience. Despite the importance of such blended emotions, most video-based emotion recognition approaches are designed to recognize single emotions only. The few approaches that have attempted to recognize blended emotions typically cannot assess the relative salience of the emotions within a blend. This limitation largely stems from the lack of datasets containing a substantial number of blended emotion samples annotated with relative salience. To address this shortcoming, we introduce BLEMORE, a novel dataset for multimodal (video, audio) blended emotion recognition that includes information on the relative salience of each emotion within a blend. BLEMORE comprises over 3,000 clips from 58 actors, performing 6 basic emotions and 10 distinct blends, where each blend has 3 different salience configurations (50/50, 70/30, and 30/70). Using this dataset, we conduct extensive evaluations of state-of-the-art video classification approaches on two blended emotion prediction tasks: (1) predicting the presence of emotions in a given sample, and (2) predicting the relative salience of emotions in a blend. Our results show that unimodal classifiers achieve up to 29% presence accuracy and 13% salience accuracy on the validation set, while multimodal methods yield clear improvements, with ImageBind + WavLM reaching 35% presence accuracy and HiCMAE 18% salience accuracy. On the held-out test set, the best models achieve 33% presence accuracy (VideoMAEv2 + HuBERT) and 18% salience accuracy (HiCMAE). In sum, the BLEMORE dataset provides a valuable resource to advancing research on emotion recognition systems that account for the complexity and significance of blended emotion expressions.

Figures (5)

• Figure 1: Examples of stills from the video recordings. The actor portrays a combination of anger and fear. Reproduced from IsraelssonBlended under CC BY 4.0.
• Figure 2: Distribution of video durations in the dataset for single and blended emotions.
• Figure 3: Structure of the BlEmoRe full dataset (train and test partition), which contains single emotions and blended emotions expressed with equal ($=$) and unequal ($<$) salience.
• Figure 4: Confusion matrix for the test set using the best overall model (VideoMAEv2 + HuBERT, Aggregation).
• Figure 5: 2D PCA projection of embeddings for the happy and sad emotions.