Implicit Modeling for Transferability Estimation of Vision Foundation Models

TL;DR

The paper tackles the challenge of transferability estimation for increasingly diverse vision foundation models. It introduces Implicit Transferability Modeling (ITM), which treats a model's transferability as a latent factor $z$ and represents the post-fine-tuning embedding as a posterior $q(\hat{\mathbf{E}}|\mathbf{E}, z)$. To keep the approach scalable, ITM employs Divide-and-Conquer Variational Approximation (DVA), partitioning the embedding space into subspaces and using a deparametric update to approximate evolution without full fine-tuning. Across a comprehensive benchmark spanning multiple architectures and pre-training strategies, ITM achieves state-of-the-art transferability estimates with superior stability and efficiency, and extends effectively to segmentation tasks. The work advances model selection for downstream tasks by offering a general, task-agnostic TE framework with broad applicability and practical impact in reducing computational cost during model deployment.

Abstract

Transferability estimation identifies the best pre-trained models for downstream tasks without incurring the high computational cost of full fine-tuning. This capability facilitates deployment and advances the pre-training and fine-tuning paradigm. However, existing methods often struggle to accurately assess transferability for emerging pre-trained models with diverse architectures, training strategies, and task alignments. In this work, we propose Implicit Transferability Modeling (ITM), a novel framework that implicitly models each model's intrinsic transferability, coupled with a Divide-and-Conquer Variational Approximation (DVA) strategy to efficiently approximate embedding space evolution. This design enables generalization across a broader range of models and downstream tasks. Extensive experiments on a comprehensive benchmark--spanning extensive training regimes and a wider variety of model types--demonstrate that ITM consistently outperforms existing methods in terms of stability, effectiveness, and efficiency.

Figures (9)

• Figure 1: Comparison between previous dynamic TE methods and the proposed ITM. (a) Previous methods simulate the entire embedding space by hand-crafted rules and estimate transferability based on the approximated target space. (b) ITM implicitly models transferability $z$ and simulates subspace evolution using a divide-and-conquer strategy for more accurate estimation.
• Figure 2: Illustration of the proposed Implicit Transferability Modeling (ITM) paradigm. (a) Ground-truth model ranking. (b) Overview of ITM, which approximates embedding space evolution via Divide-and-Conquer Variational Approximation (DVA). (c) Detailed view of DVA on a single mini-batch: the transferability $z$ is integrated into $\mathbf{E}_j$ to form the posterior condition $\mathbf{\Theta}_j$ via $\mathbf{W}_z$, which is jointly optimized with the downstream task head through the downstream objective $\mathcal{L}_{obj}$.
• Figure 3: Comparison of stability between ITM and state-of-the-art methods. The axes show the weighted Kendall’s $\tau_w$ scores of each method. Points appearing above the $y = x$ dashed line indicate cases where ITM achieves higher estimation accuracy than its counterparts.
• Figure 4: Ablation study on batch size. (a) Weighted Kendall’s $\tau_w$ across different combinations of batch size and iteration count. (b) Impact of batch size on weighted Kendall’s $\tau_w$ and running time.
• Figure 5: Comparison between ITM, recent TE methods, and ground-truth performance across ten benchmarks. Scores are normalized to $[0.3, 1]$ for clearer visualization. (a) Evaluation on MAE and CNN pre-trained models. Static TE methods (e.g., PARC parc and ETran etran) fail to generalize to MAE mae due to weak discriminative power in initial embeddings. (b) Evaluation on DINO and MAE pre-trained models. Dynamic TE methods (e.g., SFDA sfda and PED ped) overestimate MAE’s performance. In contrast, ITM generalizes well across diverse pre-trained models and provides accurate estimations.