DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis

TL;DR

DashFusion tackles multimodal sentiment analysis by jointly addressing temporal and semantic alignment through a dual-stream alignment module and enhancing feature discrimination with supervised contrastive learning. It introduces hierarchical bottleneck fusion to integrate modalities via progressively compressed bottleneck tokens, balancing accuracy and efficiency. Empirical results on CMU-MOSI, CMU-MOSEI, and CH-SIMS show state-of-the-art or competitive performance with strong ablations validating the contributions. The approach yields a robust, scalable solution for MSA with practical benefits for real-world, multi-modal sentiment understanding.

Abstract

Multimodal sentiment analysis (MSA) integrates various modalities, such as text, image, and audio, to provide a more comprehensive understanding of sentiment. However, effective MSA is challenged by alignment and fusion issues. Alignment requires synchronizing both temporal and semantic information across modalities, while fusion involves integrating these aligned features into a unified representation. Existing methods often address alignment or fusion in isolation, leading to limitations in performance and efficiency. To tackle these issues, we propose a novel framework called Dual-stream Alignment with Hierarchical Bottleneck Fusion (DashFusion). Firstly, dual-stream alignment module synchronizes multimodal features through temporal and semantic alignment. Temporal alignment employs cross-modal attention to establish frame-level correspondences among multimodal sequences. Semantic alignment ensures consistency across the feature space through contrastive learning. Secondly, supervised contrastive learning leverages label information to refine the modality features. Finally, hierarchical bottleneck fusion progressively integrates multimodal information through compressed bottleneck tokens, which achieves a balance between performance and computational efficiency. We evaluate DashFusion on three datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS. Experimental results demonstrate that DashFusion achieves state-of-the-art performance across various metrics, and ablation studies confirm the effectiveness of our alignment and fusion techniques. The codes for our experiments are available at https://github.com/ultramarineX/DashFusion.

Figures (4)

• Figure 1: The temporal misalignment and modality heterogeneity in the pipeline of multimodal sentiment analysis (MSA).
• Figure 2: The overall architecture of the DashFusion model for MSA. It consists of modality encoding, dual-stream alignment, supervised contrastive learning, and hierarchical bottleneck fusion. Features are initially encoded independently, aligned temporally and semantically, refined through supervised contrastive learning, and finally fused via hierarchical bottleneck layers to produce robust sentiment predictions.
• Figure 3: Hierarchical Bottleneck Fusion (HBF) layer architecture (right). Multi-CA (left) gathers and integrate information from different modality features through cross-modal attention (CA).
• Figure 4: Performance evaluation of DashFusion with varying numbers of bottleneck tokens on the CH-SIMS dataset. The left plot presents classification accuracy across different class settings, while the right plot illustrates regression metrics, including Mean Absolute Error (MAE) and Pearson correlation (Corr).