Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis

Ziyang Liu, Pengjunfei Chu, Shuming Dong, Chen Zhang, Mingcheng Li, Jin Wang

TL;DR

This work tackles the challenge of incomplete multimodal sentiment analysis by introducing FSRF, which jointly learns to recover missing semantics and align distributions across modalities. The DHF module decomposes each modality into modality-homogeneous, modality-heterogeneous, and noise components, enforced by contrastive and entropy-based constraints to preserve task-relevant information while suppressing noise. The DAS module enables bidirectional knowledge transfer between missing-pattern samples using the Sinkhorn distance for feature alignment and Jensen-Shannon divergence for logits alignment, integrated into a unified objective. Experiments on MOSI and MOSEI show that FSRF outperforms state-of-the-art missing-modality methods and remains competitive with complete-modality models, demonstrating strong robustness to intra- and inter-modality missingness. The proposed framework offers a principled, distribution-aware approach to semantic recovery in incomplete multimodal data, with implications for more reliable real-world MSA systems.

Abstract

In recent years, Multimodal Sentiment Analysis (MSA) has become a research hotspot that aims to utilize multimodal data for human sentiment understanding. Previous MSA studies have mainly focused on performing interaction and fusion on complete multimodal data, ignoring the problem of missing modalities in real-world applications due to occlusion, personal privacy constraints, and device malfunctions, resulting in low generalizability. To this end, we propose a Factorization-guided Semantic Recovery Framework (FSRF) to mitigate the modality missing problem in the MSA task. Specifically, we propose a de-redundant homo-heterogeneous factorization module that factorizes modality into modality-homogeneous, modality-heterogeneous, and noisy representations and design elaborate constraint paradigms for representation learning. Furthermore, we design a distribution-aligned self-distillation module that fully recovers the missing semantics by utilizing bidirectional knowledge transfer. Comprehensive experiments on two datasets indicate that FSRF has a significant performance advantage over previous methods with uncertain missing modalities.

FSRF: Factorization-guided Semantic Recovery for Incomplete Multimodal Sentiment Analysis

TL;DR

This work tackles the challenge of incomplete multimodal sentiment analysis by introducing FSRF, which jointly learns to recover missing semantics and align distributions across modalities. The DHF module decomposes each modality into modality-homogeneous, modality-heterogeneous, and noise components, enforced by contrastive and entropy-based constraints to preserve task-relevant information while suppressing noise. The DAS module enables bidirectional knowledge transfer between missing-pattern samples using the Sinkhorn distance for feature alignment and Jensen-Shannon divergence for logits alignment, integrated into a unified objective. Experiments on MOSI and MOSEI show that FSRF outperforms state-of-the-art missing-modality methods and remains competitive with complete-modality models, demonstrating strong robustness to intra- and inter-modality missingness. The proposed framework offers a principled, distribution-aware approach to semantic recovery in incomplete multimodal data, with implications for more reliable real-world MSA systems.

Abstract

In recent years, Multimodal Sentiment Analysis (MSA) has become a research hotspot that aims to utilize multimodal data for human sentiment understanding. Previous MSA studies have mainly focused on performing interaction and fusion on complete multimodal data, ignoring the problem of missing modalities in real-world applications due to occlusion, personal privacy constraints, and device malfunctions, resulting in low generalizability. To this end, we propose a Factorization-guided Semantic Recovery Framework (FSRF) to mitigate the modality missing problem in the MSA task. Specifically, we propose a de-redundant homo-heterogeneous factorization module that factorizes modality into modality-homogeneous, modality-heterogeneous, and noisy representations and design elaborate constraint paradigms for representation learning. Furthermore, we design a distribution-aligned self-distillation module that fully recovers the missing semantics by utilizing bidirectional knowledge transfer. Comprehensive experiments on two datasets indicate that FSRF has a significant performance advantage over previous methods with uncertain missing modalities.
Paper Structure (12 sections, 9 equations, 3 figures, 2 tables)

This paper contains 12 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Modality Preprocessing
  5. De-redundant Homo-Heterogeneous Factorization
  6. Distribution Alignment-based Self-Distillation
  7. Experiments
  8. Datasets and Implementation Details
  9. Comparison with State-of-the-art Methods
  10. Ablation Studies
  11. Qualitative Analysis
  12. Conclusion

Figures (3)

  • Figure 1: The overall architecture of FSRF. We propose a De-redundant Homo-Heterogeneous Factorization (DHF) mechanism and a Distributional Alignment-based Self-Distillation (DAS) module to accurately recover missing information.
  • Figure 2: Comparison results under intra-modality missingness cases on (a) MOSI and (b) MOSEI.
  • Figure 3: Representation visualization on the MOSI dataset.