Practical Transferability Estimation for Image Classification Tasks

TL;DR

This work introduces JC-NCE, a practical transferability metric for image classification that estimates transfer ease using joint correspondences learned via optimal transport. By defining a ground cost that combines sample-feature distance and label-distance information through a weighted scheme, JC-NCE computes the negative conditional entropy $-H(Y_t|Y_s)$ from the OT coupling, avoiding auxiliary tasks required by prior OTCE methods. The approach yields higher correlation with actual transfer accuracy than previous analytical metrics in both intra-dataset and inter-dataset transfer scenarios, while offering competitive computation efficiency compared to empirical transferability. Overall, JC-NCE advances robust, auxiliary-task-free transferability estimation with tangible benefits for source model selection and multi-task learning in cross-domain settings.

Abstract

Transferability estimation is an essential problem in transfer learning to predict how good the performance is when transferring a source model (or source task) to a target task. Recent analytical transferability metrics have been widely used for source model selection and multi-task learning. A major challenge is how to make transfereability estimation robust under the cross-domain cross-task settings. The recently proposed OTCE score solves this problem by considering both domain and task differences, with the help of transfer experiences on auxiliary tasks, which causes an efficiency overhead. In this work, we propose a practical transferability metric called JC-NCE score that dramatically improves the robustness of the task difference estimation in OTCE, thus removing the need for auxiliary tasks. Specifically, we build the joint correspondences between source and target data via solving an optimal transport problem with a ground cost considering both the sample distance and label distance, and then compute the transferability score as the negative conditional entropy of the matched labels. Extensive validations under the intra-dataset and inter-dataset transfer settings demonstrate that our JC-NCE score outperforms the auxiliary-task free version of OTCE for 7% and 12%, respectively, and is also more robust than other existing transferability metrics on average.

Figures (5)

• Figure 1: Transfer 40 source models (randomly generated 50-categories classification tasks, corresponding to each point in the figure) from Clipart, Painting, Quickdraw, Sketch domains to a target task (25-categories) in Real domain, which demonstrates that it is unreliable to perform source model selection according to the source model accuracy, but our JC-NCE score can predict the transfer performance more accurately.
• Figure 2: A toy example visualizes the optimal coupling between the source and target data. We can see that our JC-NCE produces a more reasonable coupling result, which ensures a better label-to-label matching than the OT-based NCE.
• Figure 3: Visualization of the correlations between the transfer accuracy and transferability scores under the challenging fixed category size setting, where all target tasks (50-categories classification, corresponding to each point in the figure) have similar complexities. Our JC-NCE score significantly outperforms the OT-based NCE score, especially as illustrated in the green circle.
• Figure 4: Analysis of $\lambda$.
• Figure 5: Computation time statistics.

Theorems & Definitions (1)

• Definition 1