Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis

TL;DR

MMCL tackles multimodal sentiment analysis by separating robust uni-modal representation learning (UMCC) from cross-modal dynamics (CMCP) via a pseudo siamese predictive network. It introduces instance- and sentiment-based contrastive losses to align and differentiate cross-modal signals while preserving modality-specific information. Empirical results on MOSI and MOSEI show state-of-the-art performance and robust behavior, including token-unaligned scenarios, supported by loss and representation analyses. Collectively, MMCL advances how uni- and cross-modal information can be shaped through contrastive learning to improve sentiment inference in multimodal data with varying alignment conditions.

Abstract

Multimodal representation learning is a challenging task in which previous work mostly focus on either uni-modality pre-training or cross-modality fusion. In fact, we regard modeling multimodal representation as building a skyscraper, where laying stable foundation and designing the main structure are equally essential. The former is like encoding robust uni-modal representation while the later is like integrating interactive information among different modalities, both of which are critical to learning an effective multimodal representation. Recently, contrastive learning has been successfully applied in representation learning, which can be utilized as the pillar of the skyscraper and benefit the model to extract the most important features contained in the multimodal data. In this paper, we propose a novel framework named MultiModal Contrastive Learning (MMCL) for multimodal representation to capture intra- and inter-modality dynamics simultaneously. Specifically, we devise uni-modal contrastive coding with an efficient uni-modal feature augmentation strategy to filter inherent noise contained in acoustic and visual modality and acquire more robust uni-modality representations. Besides, a pseudo siamese network is presented to predict representation across different modalities, which successfully captures cross-modal dynamics. Moreover, we design two contrastive learning tasks, instance- and sentiment-based contrastive learning, to promote the process of prediction and learn more interactive information related to sentiment. Extensive experiments conducted on two public datasets demonstrate that our method surpasses the state-of-the-art methods.

Figures (5)

• Figure 1: The overall architecture of our proposed MMCL framework.
• Figure 2: Illustration of Uni-Modal Contrastive Coding.
• Figure 3: Illustration of Cross-Modal Contrastive Prediction along with two contrastive learning tasks.
• Figure 4: Visualization of losses changing as training proceeds on CMU-MOSI. The values for plotting are the average losses in a constant interval of every 5 steps and $\mathcal{L}_{uniform}$ is exponentiated for plotting purposes.
• Figure 5: T-SNE hinton2008visualizing visualization of multimodal representation in the embedding space on the training set of CMU-MOSI.