Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling

Junhyeong Byeon, Jeongyeol Kim, Sejoon Lim

Abstract

Emotion recognition in in-the-wild video data remains a challenging problem due to large variations in facial appearance, head pose, illumination, background noise, and the inherently dynamic nature of human affect. Relying on a single modality, such as facial expressions or speech, is often insufficient to capture these complex emotional cues. To address this issue, we propose a multimodal emotion recognition framework for the Expression (EXPR) Recognition task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge. Our approach leverages large-scale pre-trained models, namely CLIP for visual encoding and Wav2Vec 2.0 for audio representation learning, as frozen backbone networks. To model temporal dependencies in facial expression sequences, we employ a Temporal Convolutional Network (TCN) over fixed-length video windows. In addition, we introduce a bi-directional cross-attention fusion module, in which visual and audio features interact symmetrically to enhance cross-modal contextualization and capture complementary emotional information. A lightweight classification head is then used for final emotion prediction. We further incorporate a text-guided contrastive objective based on CLIP text features to encourage semantically aligned visual representations. Experimental results on the ABAW 10th EXPR benchmark show that the proposed framework provides a strong multimodal baseline and achieves improved performance over unimodal modeling. These results demonstrate the effectiveness of combining temporal visual modeling, audio representation learning, and cross-modal fusion for robust emotion recognition in unconstrained real-world environments.

Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling

Abstract

Emotion recognition in in-the-wild video data remains a challenging problem due to large variations in facial appearance, head pose, illumination, background noise, and the inherently dynamic nature of human affect. Relying on a single modality, such as facial expressions or speech, is often insufficient to capture these complex emotional cues. To address this issue, we propose a multimodal emotion recognition framework for the Expression (EXPR) Recognition task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge. Our approach leverages large-scale pre-trained models, namely CLIP for visual encoding and Wav2Vec 2.0 for audio representation learning, as frozen backbone networks. To model temporal dependencies in facial expression sequences, we employ a Temporal Convolutional Network (TCN) over fixed-length video windows. In addition, we introduce a bi-directional cross-attention fusion module, in which visual and audio features interact symmetrically to enhance cross-modal contextualization and capture complementary emotional information. A lightweight classification head is then used for final emotion prediction. We further incorporate a text-guided contrastive objective based on CLIP text features to encourage semantically aligned visual representations. Experimental results on the ABAW 10th EXPR benchmark show that the proposed framework provides a strong multimodal baseline and achieves improved performance over unimodal modeling. These results demonstrate the effectiveness of combining temporal visual modeling, audio representation learning, and cross-modal fusion for robust emotion recognition in unconstrained real-world environments.
Paper Structure (17 sections, 10 equations, 1 figure, 1 table)

This paper contains 17 sections, 10 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Facial Expression Recognition
  4. Vision-Language Emotion Recognition
  5. Temporal Modeling
  6. Multimodal Fusion
  7. Method
  8. Feature Extraction and Prompt Generation
  9. Temporal Modeling and Audio Feature Adaptation
  10. Bi-directional Cross-Attention Fusion
  11. Temporal Pooling and Emotion Classifier
  12. Text-Guided Contrastive Learning
  13. Experiments
  14. Implementation and Evaluation Setup
  15. Evaluation Metrics
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Overview of the proposed multimodal emotion recognition framework. The model encodes video frames and audio signals using frozen CLIP and Wav2Vec 2.0 backbones, respectively. The visual features are further refined by a temporal convolutional network, while the audio features are projected through an adapter layer. The resulting multimodal representations are integrated via bi-directional cross-attention, followed by temporal pooling and an MLP classifier for expression recognition. During training, text prompts are additionally used to impose a text-guided contrastive objective.