Few-shot Adaptation of Multi-modal Foundation Models: A Survey

TL;DR

The paper surveys few-shot adaptation for multi-modal foundation models, addressing the gap between broad pre-training and fine-grained downstream tasks. It classifies existing methods into prompt-based, adapter-based, and external knowledge-based approaches, and synthesizes empirical datasets and four evaluation setups. A theoretical few-shot cross-domain generalization bound is derived, highlighting domain gap, model capacity, and sample size as key factors, and it motivates three adaptive directions: adaptive domain generalization, adaptive model selection, and adaptive knowledge utilization. The work offers a structured taxonomy, practical insights for evaluation, and guidance for future research to improve generalization in domains like medicine and remote sensing.

Abstract

Multi-modal (vision-language) models, such as CLIP, are replacing traditional supervised pre-training models (e.g., ImageNet-based pre-training) as the new generation of visual foundation models. These models with robust and aligned semantic representations learned from billions of internet image-text pairs and can be applied to various downstream tasks in a zero-shot manner. However, in some fine-grained domains like medical imaging and remote sensing, the performance of multi-modal foundation models often leaves much to be desired. Consequently, many researchers have begun to explore few-shot adaptation methods for these models, gradually deriving three main technical approaches: 1) prompt-based methods, 2) adapter-based methods, and 3) external knowledge-based methods. Nevertheless, this rapidly developing field has produced numerous results without a comprehensive survey to systematically organize the research progress. Therefore, in this survey, we introduce and analyze the research advancements in few-shot adaptation methods for multi-modal models, summarizing commonly used datasets and experimental setups, and comparing the results of different methods. In addition, due to the lack of reliable theoretical support for existing methods, we derive the few-shot adaptation generalization error bound for multi-modal models. The theorem reveals that the generalization error of multi-modal foundation models is constrained by three factors: domain gap, model capacity, and sample size. Based on this, we propose three possible solutions from the following aspects: 1) adaptive domain generalization, 2) adaptive model selection, and 3) adaptive knowledge utilization.

Figures (4)

• Figure 1: Schematic diagram of prompt-based fine-tuning adaptation methods.
• Figure 2: Schematic diagram of adapter-based fine-tuning adaptation methods.
• Figure 3: Schematic diagram of external knowledge-based fine-tuning adaptation methods.
• Figure 4: Key issues of the few-shot adaptation methods for multi-modal foundation models and corresponding solutions.

Theorems & Definitions (1)

• proof