Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusion

TL;DR

This work tackles robust multimodal sentiment analysis for image-text pairs in real-world social media where modalities can be corrupted or missing. It introduces DRF, a framework that uses feature distribution queues to drive both modality recovery (sample- and distribution-based) and quality-aware fusion, enabling unified handling of low-quality and missing modalities. Through two disruption strategies and evaluations on MVSA-S, MVSA-M, and TumEmo, DRF consistently outperforms state-of-the-art methods and remains competitive without disruption, with ablations confirming the contribution of each component. The approach advances practical robustness in MSA by linking global distribution guidance with local sample mappings for cross-modal recovery and selective fusion.

Abstract

As posts on social media increase rapidly, analyzing the sentiments embedded in image-text pairs has become a popular research topic in recent years. Although existing works achieve impressive accomplishments in simultaneously harnessing image and text information, they lack the considerations of possible low-quality and missing modalities. In real-world applications, these issues might frequently occur, leading to urgent needs for models capable of predicting sentiment robustly. Therefore, we propose a Distribution-based feature Recovery and Fusion (DRF) method for robust multimodal sentiment analysis of image-text pairs. Specifically, we maintain a feature queue for each modality to approximate their feature distributions, through which we can simultaneously handle low-quality and missing modalities in a unified framework. For low-quality modalities, we reduce their contributions to the fusion by quantitatively estimating modality qualities based on the distributions. For missing modalities, we build inter-modal mapping relationships supervised by samples and distributions, thereby recovering the missing modalities from available ones. In experiments, two disruption strategies that corrupt and discard some modalities in samples are adopted to mimic the low-quality and missing modalities in various real-world scenarios. Through comprehensive experiments on three publicly available image-text datasets, we demonstrate the universal improvements of DRF compared to SOTA methods under both two strategies, validating its effectiveness in robust multimodal sentiment analysis.

Figures (6)

• Figure 1: Brief illustration of DRF. We maintain two feature queues to approximate the feature distributions of images and texts. The distributions can estimate the contribution of each modality for fusion and provide global guidance for modality recovery, facilitating the robustness of the model to both low-quality and missing modalities.
• Figure 2: Illustration of DRF. The core of our method is the modeling of image and text feature distributions, which we approximate using the respective feature queues. After separate encoding of each modality, we first supervise two converters to learn inter-modal mapping relationships by sample-based and distribution-based recovery. Subsequently, we leverage the recovered features to expand each sample into three. Utilizing the Gaussian distribution probability, we estimate the modality qualities to decide their contributions to the fusion. Finally, we obtain the overall fused feature as the weighted sum of the features of three expanded samples and enqueue features to the queue according to their qualities.
• Figure 3: Examples of estimating image quality based on the feature distribution.
• Figure 4: Illustration of modality-fixed disruption and modality-random disruption strategies.
• Figure 5: Model performances under modality-random disruption. We report ACC scores of models under C, D, and C+D settings on MVSA-S, MVSA-M, and TumEmo.