Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network

Vrushank Ahire, Kunal Shah, Mudasir Nazir Khan, Nikhil Pakhale, Lownish Rai Sookha, M. A. Ganaie, Abhinav Dhall

TL;DR

This work tackles robust continuous emotion estimation in-the-wild by leveraging a multi-modal attention framework that fuses visual, audio, and textual cues. MAVEN employs bidirectional cross-modal attention across all modality pairs, followed by intra-modal BEiT-based refinement and a polar-coordinate VA predictor, enabling effective temporal and cross-modal information exchange. On Aff-Wild2, MAVEN achieves a state-of-the-art $CCC_{avg} = 0.3061$, outpacing the baseline and demonstrating the value of integrated cross-modal modeling with a circumplex VA representation. The approach offers practical impact for real-world affective computing applications by improving recognition in unconstrained settings and aligning predictions with psychological emotion models through $valence$ and $arousal$ expressed via $valence = I \cos(\theta)$ and $arousal = I \sin(\theta)$.

Abstract

Dynamic emotion recognition in the wild remains challenging due to the transient nature of emotional expressions and temporal misalignment of multi-modal cues. Traditional approaches predict valence and arousal and often overlook the inherent correlation between these two dimensions. The proposed Multi-modal Attention for Valence-Arousal Emotion Network (MAVEN) integrates visual, audio, and textual modalities through a bi-directional cross-modal attention mechanism. MAVEN uses modality-specific encoders to extract features from synchronized video frames, audio segments, and transcripts, predicting emotions in polar coordinates following Russell's circumplex model. The evaluation of the Aff-Wild2 dataset using MAVEN achieved a concordance correlation coefficient (CCC) of 0.3061, surpassing the ResNet-50 baseline model with a CCC of 0.22. The multistage architecture captures the subtle and transient nature of emotional expressions in conversational videos and improves emotion recognition in real-world situations. The code is available at: https://github.com/Vrushank-Ahire/MAVEN_8th_ABAW

MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network

TL;DR

This work tackles robust continuous emotion estimation in-the-wild by leveraging a multi-modal attention framework that fuses visual, audio, and textual cues. MAVEN employs bidirectional cross-modal attention across all modality pairs, followed by intra-modal BEiT-based refinement and a polar-coordinate VA predictor, enabling effective temporal and cross-modal information exchange. On Aff-Wild2, MAVEN achieves a state-of-the-art , outpacing the baseline and demonstrating the value of integrated cross-modal modeling with a circumplex VA representation. The approach offers practical impact for real-world affective computing applications by improving recognition in unconstrained settings and aligning predictions with psychological emotion models through and expressed via and .

Abstract

Dynamic emotion recognition in the wild remains challenging due to the transient nature of emotional expressions and temporal misalignment of multi-modal cues. Traditional approaches predict valence and arousal and often overlook the inherent correlation between these two dimensions. The proposed Multi-modal Attention for Valence-Arousal Emotion Network (MAVEN) integrates visual, audio, and textual modalities through a bi-directional cross-modal attention mechanism. MAVEN uses modality-specific encoders to extract features from synchronized video frames, audio segments, and transcripts, predicting emotions in polar coordinates following Russell's circumplex model. The evaluation of the Aff-Wild2 dataset using MAVEN achieved a concordance correlation coefficient (CCC) of 0.3061, surpassing the ResNet-50 baseline model with a CCC of 0.22. The multistage architecture captures the subtle and transient nature of emotional expressions in conversational videos and improves emotion recognition in real-world situations. The code is available at: https://github.com/Vrushank-Ahire/MAVEN_8th_ABAW

Paper Structure

This paper contains 36 sections, 26 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unimodal Emotion Recognition
  4. Visual-Based Approaches
  5. Audio-Based Approaches
  6. Text-Based Approaches
  7. Attention Mechanisms for Emotion Recognition
  8. Dimensional Models of Emotion
  9. Proposed Model: MAVEN
  10. Overview
  11. Visual Features
  12. Patch Embedding
  13. Shifted Window Multi-Head Self-Attention (SW-MSA)
  14. Visual Feature Output
  15. Audio Features
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Valence-Arousal Emotion Circumplex
  • Figure 2: Valence-Arousal Distribution for a video sample from the Aff-Wild2 dataset
  • Figure 3: Multi-modal Attention for Valence-Arousal Emotion Network (MAVEN) Architecture. The model processes visual, audio, and text inputs through specialized feature extractors, fuses information via cross-modal attention pathways, refines representations with bidirectional self-attention, and predicts emotions using a polar coordinate system.
  • Figure 4: Example frames from the Aff-Wild2 dataset showing facial region extraction and alignment.