Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency

TL;DR

The paper addresses scalable multimodal fusion by introducing Low-rank Multimodal Fusion (LMF) that factorizes fusion weights into modality-specific components. By leveraging a rank-$r$ decomposition and parallelizable computations, LMF avoids forming large input tensors and achieves linear scaling with the number of modalities while maintaining competitive performance on sentiment, emotion, and speaker trait tasks. Empirical results show substantial reductions in parameters and training/inference time compared to tensor-based baselines like TFN, with robust performance across rank settings. This approach yields practical benefits for deploying multimodal systems in real-world scenarios with many modalities.

Abstract

Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exploited the expressiveness of tensors for multimodal representation. However, these methods often suffer from exponential increase in dimensions and in computational complexity introduced by transformation of input into tensor. In this paper, we propose the Low-rank Multimodal Fusion method, which performs multimodal fusion using low-rank tensors to improve efficiency. We evaluate our model on three different tasks: multimodal sentiment analysis, speaker trait analysis, and emotion recognition. Our model achieves competitive results on all these tasks while drastically reducing computational complexity. Additional experiments also show that our model can perform robustly for a wide range of low-rank settings, and is indeed much more efficient in both training and inference compared to other methods that utilize tensor representations.

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

TL;DR

The paper addresses scalable multimodal fusion by introducing Low-rank Multimodal Fusion (LMF) that factorizes fusion weights into modality-specific components. By leveraging a rank- decomposition and parallelizable computations, LMF avoids forming large input tensors and achieves linear scaling with the number of modalities while maintaining competitive performance on sentiment, emotion, and speaker trait tasks. Empirical results show substantial reductions in parameters and training/inference time compared to tensor-based baselines like TFN, with robust performance across rank settings. This approach yields practical benefits for deploying multimodal systems in real-world scenarios with many modalities.

Abstract

Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exploited the expressiveness of tensors for multimodal representation. However, these methods often suffer from exponential increase in dimensions and in computational complexity introduced by transformation of input into tensor. In this paper, we propose the Low-rank Multimodal Fusion method, which performs multimodal fusion using low-rank tensors to improve efficiency. We evaluate our model on three different tasks: multimodal sentiment analysis, speaker trait analysis, and emotion recognition. Our model achieves competitive results on all these tasks while drastically reducing computational complexity. Additional experiments also show that our model can perform robustly for a wide range of low-rank settings, and is indeed much more efficient in both training and inference compared to other methods that utilize tensor representations.

Paper Structure

This paper contains 19 sections, 8 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Low-rank Multimodal Fusion
  4. Multimodal Fusion using Tensor Representations
  5. Low-rank Multimodal Fusion with Modality-Specific Factors
  6. Low-rank Weight Decomposition
  7. Efficient Low-rank Fusion Exploiting Parallel Decomposition
  8. Experimental Methodology
  9. Datasets
  10. Features
  11. Model Architecture
  12. Baseline Models
  13. Evaluation Metrics
  14. Results and Discussion
  15. Impact of Low-rank Multimodal Fusion
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Overview of our Low-rank Multimodal Fusion model structure: LMF first obtains the unimodal representation $z_a, z_v, z_l$ by passing the unimodal inputs $x_a, x_v, x_l$ into three sub-embedding networks $f_v, f_a, f_l$ respectively. LMF produces the multimodal output representation by performing low-rank multimodal fusion with modality-specific factors. The multimodal representation can be then used for generating prediction tasks.
  • Figure 2: Tensor fusion via tensor outer product
  • Figure 3: Decomposing weight tensor into low-rank factors (See Section \ref{['par:low_rank']} for details.)
  • Figure 4: The Impact of different rank settings on Model Performance: As the rank increases, the results become unstable and low rank is enough in terms of the mean absolute error.