Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal Adapter for Vision-Language Models

Dominykas Seputis, Serghei Mihailov, Soham Chatterjee, Zehao Xiao

TL;DR

This work addresses adapting large vision-language models like CLIP to downstream tasks with limited data. It introduces the Multi-Modal Adapter, a parameter-efficient module that jointly fuses text and image features through a Masked Multi-Head Attention block, aided by dimension downsampling and a fixed blending factor of $\lambda=0.2$ to combine adapted and original embeddings. Across eleven datasets in a 16-shot setting, MMA delivers competitive performance and better generalization to unseen classes, with ablations highlighting the importance of text adaptation. The approach offers a practical path for cross-modal, task-specific adaptation and provides code to facilitate reuse.

Abstract

Large pre-trained vision-language models, such as CLIP, have demonstrated state-of-the-art performance across a wide range of image classification tasks, without requiring retraining. Few-shot CLIP is competitive with existing specialized architectures that were trained on the downstream tasks. Recent research demonstrates that the performance of CLIP can be further improved using lightweight adaptation approaches. However, previous methods adapt different modalities of the CLIP model individually, ignoring the interactions and relationships between visual and textual representations. In this work, we propose Multi-Modal Adapter, an approach for Multi-Modal adaptation of CLIP. Specifically, we add a trainable Multi-Head Attention layer that combines text and image features to produce an additive adaptation of both. Multi-Modal Adapter demonstrates improved generalizability, based on its performance on unseen classes compared to existing adaptation methods. We perform additional ablations and investigations to validate and interpret the proposed approach.

Multi-Modal Adapter for Vision-Language Models

TL;DR

This work addresses adapting large vision-language models like CLIP to downstream tasks with limited data. It introduces the Multi-Modal Adapter, a parameter-efficient module that jointly fuses text and image features through a Masked Multi-Head Attention block, aided by dimension downsampling and a fixed blending factor of to combine adapted and original embeddings. Across eleven datasets in a 16-shot setting, MMA delivers competitive performance and better generalization to unseen classes, with ablations highlighting the importance of text adaptation. The approach offers a practical path for cross-modal, task-specific adaptation and provides code to facilitate reuse.

Abstract

Large pre-trained vision-language models, such as CLIP, have demonstrated state-of-the-art performance across a wide range of image classification tasks, without requiring retraining. Few-shot CLIP is competitive with existing specialized architectures that were trained on the downstream tasks. Recent research demonstrates that the performance of CLIP can be further improved using lightweight adaptation approaches. However, previous methods adapt different modalities of the CLIP model individually, ignoring the interactions and relationships between visual and textual representations. In this work, we propose Multi-Modal Adapter, an approach for Multi-Modal adaptation of CLIP. Specifically, we add a trainable Multi-Head Attention layer that combines text and image features to produce an additive adaptation of both. Multi-Modal Adapter demonstrates improved generalizability, based on its performance on unseen classes compared to existing adaptation methods. We perform additional ablations and investigations to validate and interpret the proposed approach.
Paper Structure (19 sections, 6 equations, 4 figures, 4 tables)

This paper contains 19 sections, 6 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Vision-Language Models
  4. Parameter-efficient fine-tuning (PEFT)
  5. Method
  6. Preliminary
  7. CLIP.
  8. CLIP-Adapter.
  9. Multi-Modal Adapter
  10. Dimension downsampling.
  11. Masked Multi-Head attention.
  12. Multi-Modal Adapter.
  13. Experiments and Results
  14. Few-shot Experiment Results
  15. Inspecting Role of Text Adaptation
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Illustration of Multi-Modal adapter.
  • Figure 2: Illustration of $\text{Input}_{\text{MHA}}$. Textual representation together with visual one are concatenated into one sequence that is passed thought Masked Multi-Head attention network.
  • Figure 3: Ablation studies on the impact of class subsampling. The graph below shows test accuracy results for both "base" and "new" class splits, evaluated under different class share proportions.
  • Figure 4: Illustration of training set images for CIFAR-10 after applying Gaussian noise.