DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis

TL;DR

DeepMLF tackles how depth and dedicated capacity affect multimodal fusion for sentiment analysis by introducing learnable fusion tokens appended to a pretrained language model and coupled with an audiovisual encoder. It enables deep, progressive fusion through multiple ML Blocks, where fusion tokens accumulate linguistic information and interact with audiovisual context via gated cross-attention, while the LM backbone remains largely frozen to preserve language regularities. The learning objective combines modality-specific losses, a language modelling regularizer, and embedding augmentation, yielding state-of-the-art results on MOSI, MOSEI, and SIMS and demonstrating that a small fusion-token set (roughly 8–20) and fusion depths of 5–7 layers outperform shallower designs. The approach offers practical impact by providing a scalable, cross-modal fusion framework compatible with diverse LLM backbones and by revealing a fusion curriculum that benefits multimodal sentiment analysis.

Abstract

While multimodal fusion has been extensively studied in Multimodal Sentiment Analysis (MSA), the role of fusion depth and multimodal capacity allocation remains underexplored. In this work, we position fusion depth, scalability, and dedicated multimodal capacity as primary factors for effective fusion. We introduce DeepMLF, a novel multimodal language model (LM) with learnable tokens tailored toward deep fusion. DeepMLF leverages an audiovisual encoder and a pretrained decoder LM augmented with multimodal information across its layers. We append learnable tokens to the LM that: 1) capture modality interactions in a controlled fashion and 2) preserve independent information flow for each modality. These fusion tokens gather linguistic information via causal self-attention in LM Blocks and integrate with audiovisual information through cross-attention MM Blocks. Serving as dedicated multimodal capacity, this design enables progressive fusion across multiple layers, providing depth in the fusion process. Our training recipe combines modality-specific losses and language modelling loss, with the decoder LM tasked to predict ground truth polarity. Across three MSA benchmarks with varying dataset characteristics, DeepMLF achieves state-of-the-art performance. Our results confirm that deeper fusion leads to better performance, with optimal fusion depths (5-7) exceeding those of existing approaches. Additionally, our analysis on the number of fusion tokens reveals that small token sets ($\sim$20) achieve optimal performance. We examine the importance of representation learning order (fusion curriculum) through audiovisual encoder initialization experiments. Our ablation studies demonstrate the superiority of the proposed fusion design and gating while providing a holistic examination of DeepMLF's scalability to LLMs, and the impact of each training objective and embedding regularization.

Figures (3)

• Figure 1: DeepMLF architecture overview. Audio and Visual features are being processed by a trainable AV-Encoder and then fed to the Language Model (LM) for deep multimodal fusion. The LM consists of $N$ layers where the LM Block remains frozen and the MM Block is trainable. The output of the overall architecture are audiovisual, language and fused tokens which encapsulate audiovisual, linguistic and mutlimodal sentiment information respectively. The Model Heads denote the involved objectives in out training recipe.
• Figure 2: MM Block. The fusion tokens (green) are appended at the LM input. First they accumulate linguistic information though the frozen LM Blocks ($\operatorname{CSA}$ and $\operatorname{FFW}$). Then the fusion tokens ($\mathbf{X}_f$) are fed to the $\operatorname{GCA}$ module where they are fused with audiovisual information ($\mathbf{z}$) and together with the language tokens ($\mathbf{X}_t$) are fed to the $\operatorname{FFW}$ module of the MM Block. This modular design allows for integration of multiple MM Blocks across LM layers, enabling deep fusion capabilities.
• Figure 3: Impact of the number $n_f$ of learnable fusion tokens on MOSEI. $\uparrow/\downarrow$: higher/lower is better.