MInD: Improving Multimodal Sentiment Analysis via Multimodal Information Disentanglement

TL;DR

MInD tackles modality heterogeneity in multimodal sentiment analysis by decomposing each modality into a modality-invariant component and a modality-specific component using a shared encoder plus private encoders. A novel adversarial noise pathway aligns uninformativeness in the private subspace, and a suite of constraints (information, consistency, difference, and reconstruction) plus cyclic reconstruction and noise-prediction refine the representations. The method achieves state-of-the-art or competitive results on CMU-MOSI, CMU-MOSEI, UR-FUNNY, and extends to multi-modal intent recognition, while ablations validate the importance of each component and constraint. The approach yields a simpler, more robust fusion and demonstrates the practical value of explicitly modeling uninformativeness in disentangled multi-modal representations.

Abstract

Learning effective joint representations has been a central task in multi-modal sentiment analysis. Previous works addressing this task focus on exploring sophisticated fusion techniques to enhance performance. However, the inherent heterogeneity of distinct modalities remains a core problem that brings challenges in fusing and coordinating the multi-modal signals at both the representational level and the informational level, impeding the full exploitation of multi-modal information. To address this problem, we propose the Multi-modal Information Disentanglement (MInD) method, which decomposes the multi-modal inputs into modality-invariant and modality-specific components through a shared encoder and multiple private encoders. Furthermore, by explicitly training generated noise in an adversarial manner, MInD is able to isolate uninformativeness, thus improves the learned representations. Therefore, the proposed disentangled decomposition allows for a fusion process that is simpler than alternative methods and results in improved performance. Experimental evaluations conducted on representative benchmark datasets demonstrate MInD's effectiveness in both multi-modal emotion recognition and multi-modal humor detection tasks. Code will be released upon acceptance of the paper.

Figures (2)

• Figure 1: Our proposed model. Each embedding $Z_{m}$ from the backbones is fed into a shared encoder and multiple private encoders to generate modality-invariant components $S_{m}$ and modality-specific components $P_{m}$, respectively. Meanwhile, by passing Gaussian noise $G_{m}$ to the private encoders, we align the uninformativeness within the feature subspace. This process is guided by mutual information maximization, the consistency loss and difference loss. After that, we proposed the vanilla reconstruction module, the cyclic reconstruction module and the noise prediction module to further improve the representations.
• Figure 2: We present the visualization results of noise vectors, as well as the modality-invariant and modality-specific components from distinct modalities, taking the testing set of UR-FUNNY as example. Blue: Invariant; Red: Specific; Green: Noise. MInD is able to depict different aspects of information.