Downstream Analysis of Foundational Medical Vision Models for Disease Progression

TL;DR

This work tackles disease progression prediction from longitudinal knee MRIs using pretrained medical vision foundations. It compares registration-based features from uniGradICON and segmentation-based features from SAM-Med3D, SwinUNETR, and a task-specific UNet, all routed through a simple linear probe. The results show that registration features can predict progression even without affine alignment and are robust to misalignment, while segmentation features require spatial alignment and ROI-focused preprocessing to achieve strong performance; an atlas reference and later network layers further improve signals. Overall, the study demonstrates the potential of foundation-model features for prognosis in medical imaging and provides practical guidance on preprocessing choices for progression tasks.

Abstract

Medical vision foundational models are used for a wide variety of tasks, including medical image segmentation and registration. This work evaluates the ability of these models to predict disease progression using a simple linear probe. We hypothesize that intermediate layer features of segmentation models capture structural information, while those of registration models encode knowledge of change over time. Beyond demonstrating that these features are useful for disease progression prediction, we also show that registration model features do not require spatially aligned input images. However, for segmentation models, spatial alignment is essential for optimal performance. Our findings highlight the importance of spatial alignment and the utility of foundation model features for image registration.

Figures (1)

• Figure 1: Performance of bottleneck features from sub-networks of uniGradICON using different preprocessing techniques. We observe that later layers of the networks are more robust to misaligned inputs, and cropping improves performance.