Which Model to Transfer? A Survey on Transferability Estimation

TL;DR

This survey addresses the challenge of selecting suitable pre-trained models for downstream tasks by organizing model transferability estimation (MTE) into two regimes: source-free MTE (SF-MTE) and source-dependent MTE (SD-MTE). It provides a comprehensive taxonomy of static and dynamic approaches within each regime, detailing technique families such as feature-structure, Bayesian statistics, information theory, matrix analysis, energy-based, linear, and model-embedding methods for SF-MTE, as well as distribution matching and duality-diagram frameworks for SD-MTE. The paper also synthesizes evaluation practices, studies robustness across domains, and highlights emerging trends and open problems, including the need for unified benchmarks and extensions to foundation models. Overall, it offers a structured guide for researchers and practitioners to quantify transferability efficiently and to understand the methodological landscape of MTE. The work emphasizes that effective MTE can significantly reduce computational costs by ranking candidate models without exhaustive training while guiding future research toward robust, scalable benchmarks and broader applicability across tasks.

Abstract

Transfer learning methods endeavor to leverage relevant knowledge from existing source pre-trained models or datasets to solve downstream target tasks. With the increase in the scale and quantity of available pre-trained models nowadays, it becomes critical to assess in advance whether they are suitable for a specific target task. Model transferability estimation is an emerging and growing area of interest, aiming to propose a metric to quantify this suitability without training them individually, which is computationally prohibitive. Despite extensive recent advances already devoted to this area, they have custom terminological definitions and experimental settings. In this survey, we present the first review of existing advances in this area and categorize them into two separate realms: source-free model transferability estimation and source-dependent model transferability estimation. Each category is systematically defined, accompanied by a comprehensive taxonomy. Besides, we address challenges and outline future research directions, intending to provide a comprehensive guide to aid researchers and practitioners.

Figures (3)

• Figure 1: The paradigm of model transferability estimation.
• Figure 2: Taxonomy of source-free model transferability estimation methods (Sec. \ref{['sec:mte']}).
• Figure 3: Taxonomy of source-dependent model transferability estimation methods (Sec. \ref{['sec:dte']}).