Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Geometry of Transfer: Unlocking Medical Vision Manifolds for Training-Free Model Ranking

Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang, Jiaying Zhou, Yang Liu, Qingchao Chen

TL;DR

This work proposes a novel Topology-Driven Transferability Estimation framework, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning, and significantly outperforms state-of-the-art baselines.

Abstract

The advent of large-scale self-supervised learning (SSL) has produced a vast zoo of medical foundation models. However, selecting optimal medical foundation models for specific segmentation tasks remains a computational bottleneck. Existing Transferability Estimation (TE) metrics, primarily designed for classification, rely on global statistical assumptions and fail to capture the topological complexity essential for dense prediction. We propose a novel Topology-Driven Transferability Estimation framework that evaluates manifold tractability rather than statistical overlap. Our approach introduces three components: (1) Global Representation Topology Divergence (GRTD), utilizing Minimum Spanning Trees to quantify feature-label structural isomorphism; (2) Local Boundary-Aware Topological Consistency (LBTC), which assesses manifold separability specifically at critical anatomical boundaries; and (3) Task-Adaptive Fusion, which dynamically integrates global and local metrics based on the semantic cardinality of the target task. Validated on the large-scale OpenMind benchmark across diverse anatomical targets and SSL foundation models, our approach significantly outperforms state-of-the-art baselines by around \textbf{31\%} relative improvement in the weighted Kendall, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning. The code will be made publicly available upon acceptance.

The Geometry of Transfer: Unlocking Medical Vision Manifolds for Training-Free Model Ranking

TL;DR

This work proposes a novel Topology-Driven Transferability Estimation framework, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning, and significantly outperforms state-of-the-art baselines.

Abstract

The advent of large-scale self-supervised learning (SSL) has produced a vast zoo of medical foundation models. However, selecting optimal medical foundation models for specific segmentation tasks remains a computational bottleneck. Existing Transferability Estimation (TE) metrics, primarily designed for classification, rely on global statistical assumptions and fail to capture the topological complexity essential for dense prediction. We propose a novel Topology-Driven Transferability Estimation framework that evaluates manifold tractability rather than statistical overlap. Our approach introduces three components: (1) Global Representation Topology Divergence (GRTD), utilizing Minimum Spanning Trees to quantify feature-label structural isomorphism; (2) Local Boundary-Aware Topological Consistency (LBTC), which assesses manifold separability specifically at critical anatomical boundaries; and (3) Task-Adaptive Fusion, which dynamically integrates global and local metrics based on the semantic cardinality of the target task. Validated on the large-scale OpenMind benchmark across diverse anatomical targets and SSL foundation models, our approach significantly outperforms state-of-the-art baselines by around \textbf{31\%} relative improvement in the weighted Kendall, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning. The code will be made publicly available upon acceptance.
Paper Structure (11 sections, 6 equations, 3 figures, 3 tables)

This paper contains 11 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation and Framework Overview
  4. Global Representation Topology Divergence (GRTD)
  5. Local Boundary-Aware Topological Consistency (LBTC)
  6. Task-Adaptive Topological Fusion
  7. Experiment and Results
  8. The Foundation Model Zoo and Downstream Tasks
  9. Implementation Details
  10. Results and Analysis
  11. Conclusion

Figures (3)

  • Figure 1: Left: Fine-tuning performance of diverse foundation models varies across downstream datasets, indicating that the optimal pre-trained encoder is task-dependent. Right: Visualizing transferability: Statistics vs. Topology. We compare the ground truth fine-tuning performance (Left) against rankings from the statistical metric (Middle) and our topology-driven metric (Right).
  • Figure 2: The overall framework. First, we extract multi-scale features via stratified sampling and construct topological graphs via MST. We then compute GRTD to quantify overall manifold alignment and LBTC to evaluate separability at critical anatomical boundaries. These metrics are integrated via a task-complexity gating factor, producing a final score $S_{\phi}$ that predicts fine-tuning performance without training.
  • Figure 3: Correlation between the fine-tuning performance and transferability metrics using MSF as an example. The vertical axis represents the average Dice, while the horizontal axis represents the standardized transferability metric. We want to observe a positive relationship between higher performance and higher transferability estimations.