Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval

Zijun Long, George Killick, Richard McCreadie, Gerardo Aragon Camarasa

TL;DR

The paper tackles the prohibitive cost of adapting Multimodal Large Language Models (MLLMs) to downstream vision-language tasks by introducing MultiWay-Adapter (MWA), which embeds two lightweight components—the New Knowledge Extractor and the Alignment Enhancer—into a shared MultiWay Transformer to deepen inter-modal alignment while freezing most of the backbone. This yields a plug-and-play, parameter-efficient Fine-tuning approach that maintains performance while dramatically reducing training time (up to 57%) and adding only about 2–3% trainable parameters. On BEiT-3 Base/Large with MSCOCO and Flickr30K, MWA achieves competitive or superior zero-shot and fine-tuning results, and ablations confirm both components are essential. The method demonstrates robustness to scale and offers a practical path for deploying large MLLMs in scalable image-text retrieval and related tasks, broadening real-world applicability.

Abstract

As Multimodal Large Language Models (MLLMs) grow in size, adapting them to specialized tasks becomes increasingly challenging due to high computational and memory demands. Indeed, traditional fine-tuning methods are costly, due to the need for extensive, task-specific training. While efficient adaptation methods exist that aim to reduce these costs, in practice they suffer from shallow inter-modal alignment, which severely hurts model effectiveness. To tackle these computational challenges and improve inter-modal alignment, we introduce the MultiWay-Adapter (MWA), a novel framework featuring an 'Alignment Enhancer'. This enhancer deepens inter-modal alignment, enabling high transferability with minimal tuning effort. Our experiments show that unlike prior efficient tuning approaches, MWA maintains model effectiveness, while reducing training time by up-to 57%. MWA is also lightweight, increasing model size by only 2-3% (in terms of parameters) for state-of-the-art foundation models like BEiT-3 Large. These results demonstrate that MWA provides an efficient and effective adaptation method for MLLMs, significantly broadening their applicability.

MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval

TL;DR

The paper tackles the prohibitive cost of adapting Multimodal Large Language Models (MLLMs) to downstream vision-language tasks by introducing MultiWay-Adapter (MWA), which embeds two lightweight components—the New Knowledge Extractor and the Alignment Enhancer—into a shared MultiWay Transformer to deepen inter-modal alignment while freezing most of the backbone. This yields a plug-and-play, parameter-efficient Fine-tuning approach that maintains performance while dramatically reducing training time (up to 57%) and adding only about 2–3% trainable parameters. On BEiT-3 Base/Large with MSCOCO and Flickr30K, MWA achieves competitive or superior zero-shot and fine-tuning results, and ablations confirm both components are essential. The method demonstrates robustness to scale and offers a practical path for deploying large MLLMs in scalable image-text retrieval and related tasks, broadening real-world applicability.

Abstract

As Multimodal Large Language Models (MLLMs) grow in size, adapting them to specialized tasks becomes increasingly challenging due to high computational and memory demands. Indeed, traditional fine-tuning methods are costly, due to the need for extensive, task-specific training. While efficient adaptation methods exist that aim to reduce these costs, in practice they suffer from shallow inter-modal alignment, which severely hurts model effectiveness. To tackle these computational challenges and improve inter-modal alignment, we introduce the MultiWay-Adapter (MWA), a novel framework featuring an 'Alignment Enhancer'. This enhancer deepens inter-modal alignment, enabling high transferability with minimal tuning effort. Our experiments show that unlike prior efficient tuning approaches, MWA maintains model effectiveness, while reducing training time by up-to 57%. MWA is also lightweight, increasing model size by only 2-3% (in terms of parameters) for state-of-the-art foundation models like BEiT-3 Large. These results demonstrate that MWA provides an efficient and effective adaptation method for MLLMs, significantly broadening their applicability.
Paper Structure (7 sections, 2 equations, 2 figures, 3 tables)

This paper contains 7 sections, 2 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. MultiWay-Adapter
  5. Experiments
  6. Analysis
  7. Conclusion

Figures (2)

  • Figure 1: Comparison of a MultiWay Transformer and our MultiWay-Adapter fine-tuning. MultiWay-Adapter uses a dual-component design, including New Knowledge Extractor and Alignment Enhancer. We replace the original FFN with New Knowledge Extractor: frozen branch (left) and the trainable bottleneck module (right). Moreover, we add a Alignment Enhancer upon the original FFN to enhance the inter-modal alignment.
  • Figure 2: Evaluation of different sizes of mid-dimension New Knowledge Extractor on MSCOCO.