Transferability Estimation for Semantic Segmentation Task

TL;DR

The paper tackles the problem of efficiently predicting how well a semantic segmentation model trained on a source task will transfer to a target task. It extends the OTCE framework to segmentation by constructing a pixel-level Optimal Transport problem between source and target features and deriving a transferability score from the conditional entropy $W_T = H(Y_t|Y_s)$, with OTCE defined as $OTCE = -W_T$. To manage the vast pixel-scale outputs, it adopts a sampling strategy that uses $N$ pixels and averages over $K$ repetitions (e.g., $N=10^4$, $K=10$) while solving $OT(D^{pix}_s, D^{pix}_t)$ with a cost $c(f^i_s,f^j_t) = \| f^i_s - f^j_t \|^2_2$ and entropic regularization $\epsilon=0.1$ via Sinkhorn. Empirically, OTCE shows strong correlations with actual transfer performance in intra-dataset transfers across Cityscapes sub-domains (up to $r \approx 0.768$) and in inter-dataset transfers from BDD100K and GTA5 to Cityscapes, with better performance when the source domain is closer (BDD100K). This work provides a practical, scalable tool for pre-selecting transferable source tasks/models for semantic segmentation in applications such as autonomous driving and medical imaging.

Abstract

Transferability estimation is a fundamental problem in transfer learning to predict how good the performance is when transferring a source model (or source task) to a target task. With the guidance of transferability score, we can efficiently select the highly transferable source models without performing the real transfer in practice. Recent analytical transferability metrics are mainly designed for image classification problem, and currently there is no specific investigation for the transferability estimation of semantic segmentation task, which is an essential problem in autonomous driving, medical image analysis, etc. Consequently, we further extend the recent analytical transferability metric OTCE (Optimal Transport based Conditional Entropy) score to the semantic segmentation task. The challenge in applying the OTCE score is the high dimensional segmentation output, which is difficult to find the optimal coupling between so many pixels under an acceptable computation cost. Thus we propose to randomly sample N pixels for computing OTCE score and take the expectation over K repetitions as the final transferability score. Experimental evaluation on Cityscapes, BDD100K and GTA5 datasets demonstrates that the OTCE score highly correlates with the transfer performance.

Figures (5)

• Figure 1: Illustration of OTCE score OTCE.
• Figure 2: A toy illustration of the OTCE for semantic segmentation task.
• Figure 3: Correlation between the transfer accuracy and the OTCE score under the intra-dataset transfer setting, where target tasks are four sub-domains (cities) including Cologne, Erfurt, Jena and Strasbourg selected from the Cityscapes. Each target task has 20 source models trained on other sub-domains of Cityscapes.
• Figure 4: Correlation between the transfer accuracy and the OTCE score under the inter-dataset transfer setting, where six source models are trained on BDD100K. Target tasks are four sub-domains (cities) including Cologne, Erfurt, Jena and Strasbourg selected from the Cityscapes.
• Figure 5: Correlation between the transfer accuracy and the OTCE score under the inter-dataset transfer setting, where six source models are trained on GTA5. Target tasks are four sub-domains (cities) including Cologne, Erfurt, Jena and Strasbourg selected from the Cityscapes.

Theorems & Definitions (1)

• Definition 1