Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding the Transfer Limits of Vision Foundation Models

Shiqi Huang, Yipei Wang, Natasha Thorley, Alexander Ng, Shaheer Saeed, Mark Emberton, Shonit Punwani, Veeru Kasivisvanathan, Dean Barratt, Daniel Alexander, Yipeng Hu

TL;DR

The paper investigates why vision foundation models often underperform on some downstream tasks due to misalignment between pretraining objectives and downstream vision-imaging needs, focusing on prostate MRI. It compares MAE-based ProFound and DINOv2-based ProViCNet across five tasks, introducing three MMD-based alignment metrics to quantify pretraining-downstream similarity: D2R, D2P, and D2S, and defines RPG to gauge transfer gains. The results show a strong negative correlation between misalignment (lower alignment) and transfer gains, with task-dependent patterns driven by pretraining objectives; well-aligned tasks exhibit faster convergence and larger improvements, highlighting the importance of task-aware pretraining for clinical imaging. The findings suggest that designing pretraining objectives with downstream applicability in mind can significantly enhance transfer efficiency and broaden the clinical impact of vision foundation models.

Abstract

Foundation models leverage large-scale pretraining to capture extensive knowledge, demonstrating generalization in a wide range of language tasks. By comparison, vision foundation models (VFMs) often exhibit uneven improvements across downstream tasks, despite substantial computational investment. We postulate that this limitation arises from a mismatch between pretraining objectives and the demands of downstream vision-and-imaging tasks. Pretraining strategies like masked image reconstruction or contrastive learning shape representations for tasks such as recovery of generic visual patterns or global semantic structures, which may not align with the task-specific requirements of downstream applications including segmentation, classification, or image synthesis. To investigate this in a concrete real-world clinical area, we assess two VFMs, a reconstruction-focused MAE-based model (ProFound) and a contrastive-learning-based model (ProViCNet), on five prostate multiparametric MR imaging tasks, examining how such task alignment influences transfer performance, i.e., from pretraining to fine-tuning. Our findings indicate that better alignment between pretraining and downstream tasks, measured by simple divergence metrics such as maximum-mean-discrepancy (MMD) between the same features before and after fine-tuning, correlates with greater performance improvements and faster convergence, emphasizing the importance of designing and analyzing pretraining objectives with downstream applicability in mind.

Understanding the Transfer Limits of Vision Foundation Models

TL;DR

The paper investigates why vision foundation models often underperform on some downstream tasks due to misalignment between pretraining objectives and downstream vision-imaging needs, focusing on prostate MRI. It compares MAE-based ProFound and DINOv2-based ProViCNet across five tasks, introducing three MMD-based alignment metrics to quantify pretraining-downstream similarity: D2R, D2P, and D2S, and defines RPG to gauge transfer gains. The results show a strong negative correlation between misalignment (lower alignment) and transfer gains, with task-dependent patterns driven by pretraining objectives; well-aligned tasks exhibit faster convergence and larger improvements, highlighting the importance of task-aware pretraining for clinical imaging. The findings suggest that designing pretraining objectives with downstream applicability in mind can significantly enhance transfer efficiency and broaden the clinical impact of vision foundation models.

Abstract

Foundation models leverage large-scale pretraining to capture extensive knowledge, demonstrating generalization in a wide range of language tasks. By comparison, vision foundation models (VFMs) often exhibit uneven improvements across downstream tasks, despite substantial computational investment. We postulate that this limitation arises from a mismatch between pretraining objectives and the demands of downstream vision-and-imaging tasks. Pretraining strategies like masked image reconstruction or contrastive learning shape representations for tasks such as recovery of generic visual patterns or global semantic structures, which may not align with the task-specific requirements of downstream applications including segmentation, classification, or image synthesis. To investigate this in a concrete real-world clinical area, we assess two VFMs, a reconstruction-focused MAE-based model (ProFound) and a contrastive-learning-based model (ProViCNet), on five prostate multiparametric MR imaging tasks, examining how such task alignment influences transfer performance, i.e., from pretraining to fine-tuning. Our findings indicate that better alignment between pretraining and downstream tasks, measured by simple divergence metrics such as maximum-mean-discrepancy (MMD) between the same features before and after fine-tuning, correlates with greater performance improvements and faster convergence, emphasizing the importance of designing and analyzing pretraining objectives with downstream applicability in mind.
Paper Structure (13 sections, 2 equations, 3 figures, 1 table)

This paper contains 13 sections, 2 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Pretraining and fine-tuning
  3. Pretraining
  4. Fine-tuning
  5. Experiments
  6. Downstream Tasks
  7. Task Alignment Measurement
  8. Relative Performance Gain
  9. Results and Discussion
  10. Task Alignment Determines the Extent of Gains
  11. Pretraining Objectives Shape Transfer Preference
  12. Well-Aligned Tasks Adapt Faster
  13. Conclusion

Figures (3)

  • Figure 1: a) Illustration of the pretraining-downstream task alignment using ProFound. b) Scatter plot showing ProFound performance trained from scratch (x-axis) versus fine-tuned from pretrained weights (y-axis). Color encodes task identity, and marker size reflects D2S value.
  • Figure 2: Negative correlation between task alignment (D2R and D2S) and relative performance gain (RPG) across ProFound (a-c) and ProViCNet (d-f) in five downstream tasks. Colored markers denote downstream tasks; asterisks (*) indicate DDPM-based fine-tuning.
  • Figure 3: Training loss (top) and validation metric (bottom) of ProFound over epochs across downstream tasks.