Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Transferability: A Framework for Fair and Robust Evaluation

Alireza Kazemi, Helia Rezvani, Mahsa Baktashmotlagh

TL;DR

The paper addresses the instability and limited generalizability of existing transferability estimation methods in cross-domain transfer learning. It introduces TransferTest, a benchmarking framework that supports label-free evaluation and source-dataset independence, and proposes a weight-based transferability metric using the 1-Wasserstein distance between pre- and post-tuning weight distributions. By evaluating across diverse source datasets, model complexities, and fine-tuning strategies, the authors show that their Wasserstein-based approach yields more robust rankings (measured by Kendall's tau) and achieves a 3.5% improvement in head-training scenarios. This framework and metric offer a practical, reliable tool for model selection in real-world cross-domain applications while highlighting the limitations of traditional, label-dependent transferability estimators.

Abstract

Transferability scores aim to quantify how well a model trained on one domain generalizes to a target domain. Despite numerous methods proposed for measuring transferability, their reliability and practical usefulness remain inconclusive, often due to differing experimental setups, datasets, and assumptions. In this paper, we introduce a comprehensive benchmarking framework designed to systematically evaluate transferability scores across diverse settings. Through extensive experiments, we observe variations in how different metrics perform under various scenarios, suggesting that current evaluation practices may not fully capture each method's strengths and limitations. Our findings underscore the value of standardized assessment protocols, paving the way for more reliable transferability measures and better-informed model selection in cross-domain applications. Additionally, we achieved a 3.5\% improvement using our proposed metric for the head-training fine-tuning experimental setup. Our code is available in this repository: https://github.com/alizkzm/pert_robust_platform.

Benchmarking Transferability: A Framework for Fair and Robust Evaluation

TL;DR

The paper addresses the instability and limited generalizability of existing transferability estimation methods in cross-domain transfer learning. It introduces TransferTest, a benchmarking framework that supports label-free evaluation and source-dataset independence, and proposes a weight-based transferability metric using the 1-Wasserstein distance between pre- and post-tuning weight distributions. By evaluating across diverse source datasets, model complexities, and fine-tuning strategies, the authors show that their Wasserstein-based approach yields more robust rankings (measured by Kendall's tau) and achieves a 3.5% improvement in head-training scenarios. This framework and metric offer a practical, reliable tool for model selection in real-world cross-domain applications while highlighting the limitations of traditional, label-dependent transferability estimators.

Abstract

Transferability scores aim to quantify how well a model trained on one domain generalizes to a target domain. Despite numerous methods proposed for measuring transferability, their reliability and practical usefulness remain inconclusive, often due to differing experimental setups, datasets, and assumptions. In this paper, we introduce a comprehensive benchmarking framework designed to systematically evaluate transferability scores across diverse settings. Through extensive experiments, we observe variations in how different metrics perform under various scenarios, suggesting that current evaluation practices may not fully capture each method's strengths and limitations. Our findings underscore the value of standardized assessment protocols, paving the way for more reliable transferability measures and better-informed model selection in cross-domain applications. Additionally, we achieved a 3.5\% improvement using our proposed metric for the head-training fine-tuning experimental setup. Our code is available in this repository: https://github.com/alizkzm/pert_robust_platform.
Paper Structure (17 sections, 5 equations, 2 figures, 8 tables)

This paper contains 17 sections, 5 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transferability Estimation Metrics
  4. Challenges in Transferability Estimation
  5. Robust Transferability Evaluation
  6. Preliminaries
  7. Weight-Based Transferability Metric
  8. Evaluation Criteria
  9. Experiments
  10. Experimental Setup
  11. Implementation Details
  12. Experimental Results
  13. Performance Across Source Datasets
  14. Impact of Model Complexity
  15. Effect of Fine-tuning Strategy
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Impact of model complexity on transferability metrics. The x-axis represents seven distinct model subsets, ordered by increasing parameter count and thus complexity (level 1: First models with the lowest parameters, level 5: Last models with the highest parameters, levels 2-4: intermediate complexity selections). The y-axis displays the weighted Kendall’s tau correlation between the predicted transferability rankings of different metrics and the actual transfer learning performance. Our proposed weight-based metrics (shown in green and purple) exhibit consistently higher correlation across varying model complexity levels compared to feature-based transferability estimation methods such as ETran, LogME, and SFDA.
  • Figure 2: Comparison of standard deviations across different transferability metrics. The Wasserstein metric (our proposed method) demonstrates significantly lower variance ($\sigma = 0.028$) compared to alternative metrics when evaluated on different subsets of the supervised model-hub. Lower standard deviation indicates more consistent performance across diverse model architectures and datasets.