Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis

Chunlei Meng, Ziyang Zhou, Lucas He, Xiaojing Du, Chun Ouyang, Zhongxue Gan

TL;DR

The paper addresses spatiotemporal information asymmetry in multimodal sentiment analysis by decoupling each modality into separate temporal and spatial representations before cross-modal interaction. It introduces TSDA, featuring per-modality temporal and spatial encoders, Factor-Consistent Cross-Modal Alignment, and a Gated Recouple module that adaptively fuses aligned streams while enforcing factor separation through purity, decorrelation, and an orthogonality regularizer. Empirical results on CMU-MOSI and CMU-MOSEI show TSDA achieving state-of-the-art performance across aligned and unaligned settings, with ablations confirming the necessity of each component. The approach yields more faithful, interpretable sentiment representations and demonstrates robustness to modality reliability variations, highlighting its practical impact for multimodal understanding tasks.

Abstract

Multimodal Sentiment Analysis integrates Linguistic, Visual, and Acoustic. Mainstream approaches based on modality-invariant and modality-specific factorization or on complex fusion still rely on spatiotemporal mixed modeling. This ignores spatiotemporal heterogeneity, leading to spatiotemporal information asymmetry and thus limited performance. Hence, we propose TSDA, Temporal-Spatial Decouple before Act, which explicitly decouples each modality into temporal dynamics and spatial structural context before any interaction. For every modality, a temporal encoder and a spatial encoder project signals into separate temporal and spatial body. Factor-Consistent Cross-Modal Alignment then aligns temporal features only with their temporal counterparts across modalities, and spatial features only with their spatial counterparts. Factor specific supervision and decorrelation regularization reduce cross factor leakage while preserving complementarity. A Gated Recouple module subsequently recouples the aligned streams for task. Extensive experiments show that TSDA outperforms baselines. Ablation analysis studies confirm the necessity and interpretability of the design.

Temporal-Spatial Decouple before Act: Disentangled Representation Learning for Multimodal Sentiment Analysis

TL;DR

The paper addresses spatiotemporal information asymmetry in multimodal sentiment analysis by decoupling each modality into separate temporal and spatial representations before cross-modal interaction. It introduces TSDA, featuring per-modality temporal and spatial encoders, Factor-Consistent Cross-Modal Alignment, and a Gated Recouple module that adaptively fuses aligned streams while enforcing factor separation through purity, decorrelation, and an orthogonality regularizer. Empirical results on CMU-MOSI and CMU-MOSEI show TSDA achieving state-of-the-art performance across aligned and unaligned settings, with ablations confirming the necessity of each component. The approach yields more faithful, interpretable sentiment representations and demonstrates robustness to modality reliability variations, highlighting its practical impact for multimodal understanding tasks.

Abstract

Multimodal Sentiment Analysis integrates Linguistic, Visual, and Acoustic. Mainstream approaches based on modality-invariant and modality-specific factorization or on complex fusion still rely on spatiotemporal mixed modeling. This ignores spatiotemporal heterogeneity, leading to spatiotemporal information asymmetry and thus limited performance. Hence, we propose TSDA, Temporal-Spatial Decouple before Act, which explicitly decouples each modality into temporal dynamics and spatial structural context before any interaction. For every modality, a temporal encoder and a spatial encoder project signals into separate temporal and spatial body. Factor-Consistent Cross-Modal Alignment then aligns temporal features only with their temporal counterparts across modalities, and spatial features only with their spatial counterparts. Factor specific supervision and decorrelation regularization reduce cross factor leakage while preserving complementarity. A Gated Recouple module subsequently recouples the aligned streams for task. Extensive experiments show that TSDA outperforms baselines. Ablation analysis studies confirm the necessity and interpretability of the design.
Paper Structure (13 sections, 6 equations, 4 figures, 2 tables)

This paper contains 13 sections, 6 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Task Formulation
  4. Temporal and Spatial Disentanglement
  5. Factor-Consistent Cross-Modal Alignment
  6. Gated Recouple
  7. Training Objective
  8. Experiments
  9. Experimental Settings
  10. Comparison with State-of-the-art Methods
  11. Ablation Studies
  12. Visualization Analysis
  13. Conclusion

Figures (4)

  • Figure 1: Architecture illustration of TSDA, which decouples temporal and spatial factors, aligns them with FCCA, and adaptively fuses them through the Gated Recouple Module.
  • Figure 2: T-SNE visualization on MOSI. Red indicates stronger positive sentiment. TSDA yields the best structure.
  • Figure 3: Regularization curves during MOSI training. Similar trends are observed on MOSEI.
  • Figure 4: Sensitivity of TSDA to $\alpha$, $\beta$, and $\gamma$ on benchmarks. Performance remains stable across values.