HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

TL;DR

HiCMAE tackles data scarcity in Audio-Visual Emotion Recognition by pre-training on large unlabeled AV data with masked reconstruction and hierarchical cross-modal contrastive learning. It introduces a three-pronged design—hierarchical skip connections, hierarchical cross-modal contrastive learning, and hierarchical feature fusion—that guides intermediate-layer representations and improves cross-modal fusion. Pre-trained on VoxCeleb2, HiCMAE achieves state-of-the-art results across nine datasets for both categorical and dimensional AVER tasks, while maintaining relatively small models and efficient computation. This work demonstrates the power of hierarchical self-supervision for scalable, annotation-efficient AV emotion representation learning with strong practical impact for affective AI systems.

Abstract

Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models will be publicly available at https://github.com/sunlicai/HiCMAE.

Figures (10)

• Figure 1: Several samples selected from the MAFW liu2022mafw dataset.
• Figure 2: Comparison with state-of-the-art audio-visual methods on 9 datasets. We present Pearson correlation coefficient on Werewolf-XL zhang2021werewolf and weighted F1-score on AVCAffe sarkar2022avcaffe and MER-MULTI lian2023mer. For other datasets, we show weighted average recall (WAR).
• Figure 3: The overall pre-training pipeline of HiCMAE. HiCMAE mainly adopts an asymmetric encoder-decoder architecture with hierarchical skip connections in between for masked audio-visual reconstruction. Besides, hierarchical cross-modal contrastive learning is employed at intermediate audio-visual encoder layers to reduce the modality gap in a progressive manner and facilitate cross-modal fusion in subsequent layers.
• Figure 4: The illustration of hierarchical skip connections between the encoder and decoder (taking the video modality as an example).
• Figure 5: Different types of information flow in cross-modal fusion encoder.