A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter

TL;DR

Wander addresses the challenge of efficient multimodal transfer learning across any number of modalities by introducing a low-rank, token-level fusion mechanism based on outer-product fusion and CP decomposition. It augments pretrained backbones with a down-projection, sequence fusion, and a residual block to enable fine-grained interactions without updating the backbone. Empirical results on four datasets spanning 2–7 modalities show Wander consistently surpasses existing efficient-transfer baselines while using substantially fewer trainable parameters, approaching full fine-tuning performance in some cases. The work provides a scalable, universal approach for adapting large multimodal models to diverse downstream tasks with strong practical impact.

Abstract

Efficient transfer learning methods such as adapter-based methods have shown great success in unimodal models and vision-language models. However, existing methods have two main challenges in fine-tuning multimodal models. Firstly, they are designed for vision-language tasks and fail to extend to situations where there are more than two modalities. Secondly, they exhibit limited exploitation of interactions between modalities and lack efficiency. To address these issues, in this paper, we propose the loW-rank sequence multimodal adapter (Wander). We first use the outer product to fuse the information from different modalities in an element-wise way effectively. For efficiency, we use CP decomposition to factorize tensors into rank-one components and achieve substantial parameter reduction. Furthermore, we implement a token-level low-rank decomposition to extract more fine-grained features and sequence relationships between modalities. With these designs, Wander enables token-level interactions between sequences of different modalities in a parameter-efficient way. We conduct extensive experiments on datasets with different numbers of modalities, where Wander outperforms state-of-the-art efficient transfer learning methods consistently. The results fully demonstrate the effectiveness, efficiency and universality of Wander.

Figures (5)

• Figure 1: The difference between vector fusion and sequence fusion in their original outer product form. Sequence fusion enables token-level interactions between modalities. We take three modalities as an example.
• Figure 2: The illustration of low-rank sequence fusion. We use three modalities as an example. $\circ$ denotes element-wise multiplication.
• Figure 3: The overall architecture of Wander and its integration with the pre-trained model. Left: We add Wander to the pre-trained model for fine-tuning. Right: Wander consists of a linear down-projection layer (Down), a nonlinear function, sequence fusion and a skip connection.
• Figure 4: The impact of $d$ on the performance on IEMOCAP.
• Figure 5: The impact of the rank of CP decomposition on the performance. We report binary accuracy for UPMC-Food 101, CMU-MOSI and IEMOCAP and R@10 for MSRVTT.