Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains

TL;DR

This work tackles the problem of transferring rich knowledge from large-scale vision pretraining to data-limited scientific domains. It introduces CLAdapter, a Cluster Attention Adapter with a unified interface and a staged fine-tuning strategy that personalizes feature transformation via cluster-centered attention and learned transformations, enabling effective cross-domain adaptation for 2D, 3D, CNN, and Transformer backbones. Across 10 diverse downstream datasets, CLAdapter delivers state-of-the-art results, demonstrating strong generalization to ID and OOD settings, with notable efficiency and compatibility advantages over existing fine-tuning methods. The approach offers practical impact for deploying foundation vision models in scientific and industrial domains with limited labeled data, while acknowledging future work in extending to detection and segmentation tasks.

Abstract

In the big data era, the computer vision field benefits from large-scale datasets such as LAION-2B, LAION-400M, and ImageNet-21K, Kinetics, on which popular models like the ViT and ConvNeXt series have been pre-trained, acquiring substantial knowledge. However, numerous downstream tasks in specialized and data-limited scientific domains continue to pose significant challenges. In this paper, we propose a novel Cluster Attention Adapter (CLAdapter), which refines and adapts the rich representations learned from large-scale data to various data-limited downstream tasks. Specifically, CLAdapter introduces attention mechanisms and cluster centers to personalize the enhancement of transformed features through distribution correlation and transformation matrices. This enables models fine-tuned with CLAdapter to learn distinct representations tailored to different feature sets, facilitating the models' adaptation from rich pre-trained features to various downstream scenarios effectively. In addition, CLAdapter's unified interface design allows for seamless integration with multiple model architectures, including CNNs and Transformers, in both 2D and 3D contexts. Through extensive experiments on 10 datasets spanning domains such as generic, multimedia, biological, medical, industrial, agricultural, environmental, geographical, materials science, out-of-distribution (OOD), and 3D analysis, CLAdapter achieves state-of-the-art performance across diverse data-limited scientific domains, demonstrating its effectiveness in unleashing the potential of foundation vision models via adaptive transfer. Code is available at https://github.com/qklee-lz/CLAdapter.

Figures (6)

• Figure 1: Overview of the proposed CLAdapter. CLAdapter refines and adapts the rich representations learned from large-scale data to diverse data-limited scientific downstream tasks, achieving state-of-the-art performance across diverse fields on 10 datasets.
• Figure 2: Overview of the CLAdapter. It utilizes large-scale pre-training to enhance various data-limited downstream tasks. The unified interface design and SFT fine-tuning strategy allow CLAdapter to integrate with mainstream pre-trained models and form an efficient fine-tuning paradigm.
• Figure 3: Performance comparison of CLAdapter against SOTA methods across various application domain datasets. The bar graph illustrates the average scores achieved by the CLAdapter and SOTA on each dataset, with the red fold line indicating the percentage improvement offered by the CLAdapter.
• Figure 4: Efficiency comparison of fine-tuning methods. CLAdapter achieves significant performance improvement with fewer training epochs and parameters, indicating both high effectiveness and efficiency. Note that standard augmentations are only applied to the training set to mitigate overfitting but not to the validation set, which results in lower validation loss than training loss.
• Figure 5: The $t$-SNE visualizations demonstrating class separability and compactness. The comparative analysis highlights the enhanced discriminability of features via CLAdapter.