Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixture of Disentangled Experts with Missing Modalities for Robust Multimodal Sentiment Analysis

Xiang Li, Xiaoming Zhang, Dezhuang Miao, Xianfu Cheng, Dawei Li, Honggui Han, Zhoujun Li

TL;DR

DERL addresses robust multimodal sentiment analysis with missing modalities by integrating a disentangled representation learning scheme within a mixture-of-experts framework. It introduces three core modules—Hybrid Expert Disentanglement, Multi-Level Collaborative Reconstruction, and Modality Routing and Fusion—to adaptively factorize private and shared cues, recover degraded semantics, and perform importance-aware fusion. Empirical results on MOSI and MOSEI show DERL outperforms state-of-the-art baselines under both intra- and inter-modal missingness, with notable gains in classification F1 and regression MAE across varying missing rates. The work demonstrates that modular disentanglement combined with multi-view reconstruction yields robust, efficient representations that are resilient to realistic noise and partial observations, advancing practical applicability of MSA in imperfect real-world data. Limitations include predefined missing patterns, suggesting future work on broader missingness scenarios and real-world deployment considerations.

Abstract

Multimodal Sentiment Analysis (MSA) integrates multiple modalities to infer human sentiment, but real-world noise often leads to missing or corrupted data. However, existing feature-disentangled methods struggle to handle the internal variations of heterogeneous information under uncertain missingness, making it difficult to learn effective multimodal representations from degraded modalities. To address this issue, we propose DERL, a Disentangled Expert Representation Learning framework for robust MSA. Specifically, DERL employs hybrid experts to adaptively disentangle multimodal inputs into orthogonal private and shared representation spaces. A multi-level reconstruction strategy is further developed to provide collaborative supervision, enhancing both the expressiveness and robustness of the learned representations. Finally, the disentangled features act as modality experts with distinct roles to generate importance-aware fusion results. Extensive experiments on two MSA benchmarks demonstrate that DERL outperforms state-of-the-art methods under various missing-modality conditions. For instance, our method achieves improvements of 2.47% in Acc-2 and 2.25% in MAE on MOSI under intra-modal missingness.

Mixture of Disentangled Experts with Missing Modalities for Robust Multimodal Sentiment Analysis

TL;DR

DERL addresses robust multimodal sentiment analysis with missing modalities by integrating a disentangled representation learning scheme within a mixture-of-experts framework. It introduces three core modules—Hybrid Expert Disentanglement, Multi-Level Collaborative Reconstruction, and Modality Routing and Fusion—to adaptively factorize private and shared cues, recover degraded semantics, and perform importance-aware fusion. Empirical results on MOSI and MOSEI show DERL outperforms state-of-the-art baselines under both intra- and inter-modal missingness, with notable gains in classification F1 and regression MAE across varying missing rates. The work demonstrates that modular disentanglement combined with multi-view reconstruction yields robust, efficient representations that are resilient to realistic noise and partial observations, advancing practical applicability of MSA in imperfect real-world data. Limitations include predefined missing patterns, suggesting future work on broader missingness scenarios and real-world deployment considerations.

Abstract

Multimodal Sentiment Analysis (MSA) integrates multiple modalities to infer human sentiment, but real-world noise often leads to missing or corrupted data. However, existing feature-disentangled methods struggle to handle the internal variations of heterogeneous information under uncertain missingness, making it difficult to learn effective multimodal representations from degraded modalities. To address this issue, we propose DERL, a Disentangled Expert Representation Learning framework for robust MSA. Specifically, DERL employs hybrid experts to adaptively disentangle multimodal inputs into orthogonal private and shared representation spaces. A multi-level reconstruction strategy is further developed to provide collaborative supervision, enhancing both the expressiveness and robustness of the learned representations. Finally, the disentangled features act as modality experts with distinct roles to generate importance-aware fusion results. Extensive experiments on two MSA benchmarks demonstrate that DERL outperforms state-of-the-art methods under various missing-modality conditions. For instance, our method achieves improvements of 2.47% in Acc-2 and 2.25% in MAE on MOSI under intra-modal missingness.
Paper Structure (39 sections, 19 equations, 4 figures, 7 tables)

This paper contains 39 sections, 19 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robust Multimodal Sentiment Analysis
  4. Disentangled Multimodal Learning
  5. Mixture-of-Experts
  6. Methodology
  7. Problem Formulation
  8. Framework Overview
  9. Multimodal Input and Encoding
  10. Random Input Missing
  11. Unified Transformer Encoding
  12. Hybrid Expert Disentanglement
  13. Multi-Level Collaborative Reconstruction
  14. Input-Level Reconstruction
  15. Disentanglement-Level Reconstruction
  16. ...and 24 more sections

Figures (4)

  • Figure 1: Overview of the proposed DERL framework. White blocks denote complete inputs, while dark gray blocks indicate missing ones. After multimodal encoding, DERL incorporates three core components to handle incomplete modalities: ① Hybrid Expert Disentanglement (HED), ② Multi-Level Collaborative Reconstruction (MLCR), and ③ Modality Routing and Fusion (MRF).
  • Figure 2: Performance trends of representative models across varying missing rates on MOSI and MOSEI datasets.
  • Figure 3: Sensitivity analysis of the number of private and shared experts on model performance on MOSI.
  • Figure 4: Seven-class confusion matrices of DERL and DMD under different missing rates on MOSI. The results from left to right correspond to $r = 0.1$, $r = 0.5$, and $r = 0.9$.