Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition

TL;DR

This paper tackles dimensional emotion recognition by leveraging rich intra- and inter-modal cues across audio, visual, and text using Recursive Joint Cross-Modal Attention (RJCMA). RJCMA builds a joint audio-visual-text representation and computes cross-correlations $\boldsymbol C_a$, $\boldsymbol C_v$, and $\boldsymbol C_t$ to generate modality-specific attention maps, with recursive refinement over $l$ iterations and temporal modeling via Temporal Convolutional Networks. The approach fuses the attended features into a multimodal representation for regression, optimized with a CCC-based loss $\mathcal{L}=1-\rho_c$, and evaluated on Affwild2, achieving competitive CCCs (validation valence/arousal around $0.455/0.652$; test around $0.542/0.619$) and rank in the ABAW competition. Experimental results demonstrate that recursive fusion and three-modality interaction yield substantial improvements over baselines, highlighting the practical impact for robust, real-time affective computing in-the-wild.

Abstract

Though multimodal emotion recognition has achieved significant progress over recent years, the potential of rich synergic relationships across the modalities is not fully exploited. In this paper, we introduce Recursive Joint Cross-Modal Attention (RJCMA) to effectively capture both intra- and inter-modal relationships across audio, visual, and text modalities for dimensional emotion recognition. In particular, we compute the attention weights based on cross-correlation between the joint audio-visual-text feature representations and the feature representations of individual modalities to simultaneously capture intra- and intermodal relationships across the modalities. The attended features of the individual modalities are again fed as input to the fusion model in a recursive mechanism to obtain more refined feature representations. We have also explored Temporal Convolutional Networks (TCNs) to improve the temporal modeling of the feature representations of individual modalities. Extensive experiments are conducted to evaluate the performance of the proposed fusion model on the challenging Affwild2 dataset. By effectively capturing the synergic intra- and inter-modal relationships across audio, visual, and text modalities, the proposed fusion model achieves a Concordance Correlation Coefficient (CCC) of 0.585 (0.542) and 0.674 (0.619) for valence and arousal respectively on the validation set(test set). This shows a significant improvement over the baseline of 0.240 (0.211) and 0.200 (0.191) for valence and arousal, respectively, in the validation set (test set), achieving second place in the valence-arousal challenge of the 6th Affective Behavior Analysis in-the-Wild (ABAW) competition.

Figures (2)

• Figure 1: Illustration of the proposed framework with the recursive joint cross-modal attention. Best viewed in color.
• Figure 2: Joint Cross-Modal Attention blocks of audio, visual and text modalities