Unsupervised and Source-Free Ranking of Biomedical Segmentation Models

TL;DR

The work introduces a zero-shot, source-free transferability estimator (CTE) to rank biomedical segmentation models without target labels. By measuring prediction consistency under input- and feature-space perturbations, CTE provides hard (NHD) and soft (EI) variants that correlate strongly with target performance for semantic segmentation, and via Adapted Rand Score for instance segmentation. Extensive experiments across EM mitochondria, LM nuclei/cells, and CBCT ToothFairy2 demonstrate that CTE outperforms unsupervised Transfer Score and supervised baselines, remains robust after unsupervised domain adaptation, and enables effective ranking with only two inference passes. The method supports practical reuse of pre-trained models in biomedical settings, though it assumes task alignment and may require refinements for mode-switching in generalist models and boundary-sensitive metrics.

Abstract

Model transfer presents a solution to the challenges of segmentation in the biomedical community, where the immense cost of data annotation is a major bottleneck in the use of deep learning. At the same time, hundreds of models get trained on biomedical data, submitted to challenges, and posted in model zoos and repositories. A major hurdle to wider adoption of pre-trained models lies in the lack of methods for best model selection. While such methods have been proposed for classification models, semantic and instance segmentation model ranking remain largely unaddressed, especially in a practically important setting where no labels are available on the target dataset. Similarly, if unsupervised domain adaptation is used, practitioners are faced with the task of selecting the best adapted model without target domain labels. Building on previous work linking model generalisation and consistency under perturbation, we propose the first unsupervised and source-free transferability estimator for semantic and instance segmentation tasks. We evaluate on multiple segmentation problems across biomedical imaging, finding a strong correlation between the rankings based on our estimator and rankings based on target dataset performance.

Figures (6)

• Figure 1: Unsupervised consistency-based model ranking.
• Figure 2: Comparison of classification and semantic segmentation transferability metric results.
• Figure 3: Semantic segmentation EPFL target correlation.
• Figure 4: Instance segmentation Covid_IF target correlation.
• Figure 5: Cells vs Nuclei Prediction of Cellpose-SAM.