A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

TL;DR

The paper tackles dimensional emotion recognition by fusing facial and vocal modalities through a novel joint cross-attentional AV fusion that models both intra- and inter-modal relationships. By constructing a joint A-V representation and using cross-correlation-based attention, the method enables each modality to attend to itself and the other modality, improving saliency extraction for valence and arousal prediction. Extensive ablations on Affwild2 show the approach outperforms traditional fusion strategies and vanilla cross-attention, with I3D+TCN providing strong temporal modeling for the visual stream and spectrogram-based ResNet18 improving audio features; results are competitive with state-of-the-art without external data. The work offers a cost-effective, robust fusion framework for in-the-wild emotional state estimation and provides publicly available code for replication and extension.

Abstract

Multimodal emotion recognition has recently gained much attention since it can leverage diverse and complementary relationships over multiple modalities (e.g., audio, visual, biosignals, etc.), and can provide some robustness to noisy modalities. Most state-of-the-art methods for audio-visual (A-V) fusion rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. In this paper, we focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. Specifically, we propose a joint cross-attention model that relies on the complementary relationships to extract the salient features across A-V modalities, allowing for accurate prediction of continuous values of valence and arousal. The proposed fusion model efficiently leverages the inter-modal relationships, while reducing the heterogeneity between the features. In particular, it computes the cross-attention weights based on correlation between the combined feature representation and individual modalities. By deploying the combined A-V feature representation into the cross-attention module, the performance of our fusion module improves significantly over the vanilla cross-attention module. Experimental results on validation-set videos from the AffWild2 dataset indicate that our proposed A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches. The code is available on GitHub: https://github.com/praveena2j/JointCrossAttentional-AV-Fusion.

Figures (2)

• Figure 1: The valence-arousal space.
• Figure 2: Joint cross-attention model proposed for A-V fusion (training mode).