Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis

TL;DR

The paper addresses multi-modal sentiment analysis by injecting explanations and temporal alignment into fusion across text, audio, and video. It introduces TEXT, a text-routed sparse mixture-of-experts model that uses multi-modal large language models to generate explanations, aligns audio and video with these explanations via a cross-attention mechanism, and employs a temporality-aware block to fuse modalities before a gate-fusion classifier outputs sentiment scores. TEXT achieves state-of-the-art performance across MOSI, MOSEI, CH-SIMS, and CH-SIMSv2, notably reducing MAE on CH-SIMS to 0.353 (approximately 13.5% better than baselines) and delivering strong Acc and F1 metrics on several datasets. The work highlights the value of explanations and temporal synchronization in robust MSA and suggests avenues to reduce accumulated errors from multiple MLLMs and extend the approach to multilingual settings.

Abstract

Human-interaction-involved applications underscore the need for Multi-modal Sentiment Analysis (MSA). Although many approaches have been proposed to address the subtle emotions in different modalities, the power of explanations and temporal alignments is still underexplored. Thus, this paper proposes the Text-routed sparse mixture-of-Experts model with eXplanation and Temporal alignment for MSA (TEXT). TEXT first augments explanations for MSA via Multi-modal Large Language Models (MLLM), and then novelly aligns the epresentations of audio and video through a temporality-oriented neural network block. TEXT aligns different modalities with explanations and facilitates a new text-routed sparse mixture-of-experts with gate fusion. Our temporal alignment block merges the benefits of Mamba and temporal cross-attention. As a result, TEXT achieves the best performance cross four datasets among all tested models, including three recently proposed approaches and three MLLMs. TEXT wins on at least four metrics out of all six metrics. For example, TEXT decreases the mean absolute error to 0.353 on the CH-SIMS dataset, which signifies a 13.5% decrement compared with recently proposed approaches.

Figures (9)

• Figure 1: An example of MSA from MOSI. On the left side, from top to bottom, are the textual, visual, and audio modalities of the short video. The corresponding modalities with MLLM-generated explanations are on the right. On the left side, only text can correctly predict the polarity, with a predicted score of 2.500. As a comparison, the label of this example is 1.400, and the TEXT's prediction is 1.390. With explanations, both video and audio modalities can correctly predict the polarity.
• Figure 2: Two temporal alignment block designs: (a) TCA, and (b) Mamba. Legends are also applicable for Fig. \ref{['fig:expAlig']}$\sim$\ref{['fig:tempAlig']}. In (a), $F_v$, $F_a$, and $F_t$ represent the input vectors to each module, while $F’_v$, $F’_a$, and $F’_t$ represent the output vectors of the module. In (b), $\otimes$ represents nonlinearity.
• Figure 3: The architecture of TEXT. From top to bottom, TEXT comprises six functional modules (①$\sim$⑥). Its sequential processing workflow centers on three core components: ① the Gate Fusion Classifier Module, which performs the final decision-making; ② the Text-Routed SMoE Module, designed to model cross-modal interactions; and ③ the Temporal Alignment Module, responsible for synchronizing audio and video streams. In addition, three uni-modal feature extraction modules (④$\sim$⑥) operate in parallel. Notably, both the Audio Feature Extraction Module (⑤) and the Video Feature Extraction Module (⑥) incorporate an Explanation Alignment Block. All modalities are processed using pre-trained encoders: textual data (including explanation annotations) is encoded with BERT, audio signals are extracted using Librosa, and visual information—derived from video frames—is processed via OpenFace.
• Figure 4: The procedure of explanation generation.
• Figure 5: Explanation aligning block.