Efficient Transferability Assessment for Selection of Pre-trained Detectors

TL;DR

This work tackles the practical problem of efficiently predicting the transferability of pre-trained detectors to downstream tasks, avoiding expensive brute-force fine-tuning. It introduces a detector transferability benchmark with 33 detectors across six diverse tasks and develops a unified, gradient-free assessment framework (Det-LogME) that extends LogME to object detection via pyramid feature matching, unified sub-task evaluation, and an IoU-based complement. The proposed Det-LogME family achieves strong ranking performance across multi-scale, multi-class detection tasks, delivering about 32× wall-clock time speedup and 19× memory savings over brute-force fine-tuning while outperforming state-of-the-art baselines. This approach enables practical, scalable model selection for detectors in varied domains, including general, driving, aerial, and medical imaging, and offers a foundation for future detector-specific transferability research. The work emphasizes the importance of multi-task, multi-scale consideration in transferability assessment and demonstrates tangible gains in both efficiency and ranking robustness.

Abstract

Large-scale pre-training followed by downstream fine-tuning is an effective solution for transferring deep-learning-based models. Since finetuning all possible pre-trained models is computational costly, we aim to predict the transferability performance of these pre-trained models in a computational efficient manner. Different from previous work that seek out suitable models for downstream classification and segmentation tasks, this paper studies the efficient transferability assessment of pre-trained object detectors. To this end, we build up a detector transferability benchmark which contains a large and diverse zoo of pre-trained detectors with various architectures, source datasets and training schemes. Given this zoo, we adopt 7 target datasets from 5 diverse domains as the downstream target tasks for evaluation. Further, we propose to assess classification and regression sub-tasks simultaneously in a unified framework. Additionally, we design a complementary metric for evaluating tasks with varying objects. Experimental results demonstrate that our method outperforms other state-of-the-art approaches in assessing transferability under different target domains while efficiently reducing wall-clock time 32$\times$ and requires a mere 5.2\% memory footprint compared to brute-force fine-tuning of all pre-trained detectors.

Figures (4)

• Figure 1: Illustration for selection of pre-trained detectors. The selection is performed by efficient transferability assessment.
• Figure 2: The overview of efficient transferability assessment framework for pre-trained detection models. We build a challenging setting contains various pre-trained object detectors. Based on this challenging setting, we design a pyramid feature matching scheme to handle objects with various sizes and expand the bounding box matrix $\boldsymbol{B}^{cen}$ according to class label matrix $\boldsymbol{C}$ to the unified label matrix $\boldsymbol{Y}^u$ for evaluation. We estimate the maximum evidence $p(\boldsymbol{Y}^u|\boldsymbol{F})$, which indicates the compatibility between the object features $\boldsymbol{F}$ and unified labels $\boldsymbol{Y}^u$. Further, considering IoU as an important metric in object detection, we supply IoU between the predicted bounding boxes and ground truth ones as a complementary term for transferability assessment of detection model.
• Figure 3: Comparison of ranking scores. The plots illustrate ground-truth fine-tuning performance $\{g_n\}_{n=1}^N$ (x-axis), ranking scores (y-axis), and Weighted Kendall’s coefficient $\tau_w$ for 33 pre-trained detectors on 3 out of 6 target datasets.
• Figure 4: Effects of different weights $\mu$ for complementary IoU metric in Det-LogME. The first blue marker indicates Det-LogME without IoU measurement (degrades to U-LogME).