A Self-Supervised Paradigm for Data-Efficient Medical Foundation Model Pre-training: V-information Optimization Framework

TL;DR

This work addresses data inefficiency in self-supervised pre-training of medical foundation models by introducing a V-information–based framework for data selection. It formalizes the objective as maximizing $I_{\mathcal{V}}(X \to Y) = H_{\mathcal{V}}(Y|\varnothing) - H_{\mathcal{V}}(Y|X)$, which is achieved by selecting harder samples to reduce $H_{\mathcal{V}}(Y|X)$ and by increasing diversity to raise $H_{\mathcal{V}}(Y|\varnothing)$. The OptiDEL method operationalizes this through SAM-based patch extraction, margin-driven hard-sample selection, and synthesis of diverse training images, with SSL pre-training via MAE and SimCLR. Across eight medical segmentation datasets, OptiDEL consistently outperforms state-of-the-art data-effective learning methods, achieving up to 6.2% higher mIoU with only 5% of the pre-training data and averaging a 4.7% mIoU gain using 20x less data, highlighting practical data-efficiency gains for medical imaging applications.

Abstract

Self-supervised pre-training medical foundation models on large-scale datasets demonstrate exceptional performance. Recent research challenges this common paradigm by introducing data-effective learning approaches, demonstrating that merely increasing pre-training data volume does not necessarily improve model performance. However, current methods still have unclear standards and the underlying theoretical foundation remains unknown. In this paper, as the first attempt to address this limitation, we introduce V-information into self-supervised pre-training of foundation models to provide a theoretical foundation for sample selection. Our derivation confirms that by optimizing V-information, sample selection can be framed as an optimization problem where choosing diverse and challenging samples enhances model performance even under limited training data. Under this guidance, we develop an optimized data-effective learning method (OptiDEL) to optimize V-information in real-world medical domains by generating more diverse and harder samples. We compare the OptiDEL method with state-of-the-art approaches finding that OptiDEL consistently outperforms existing approaches across eight different datasets, with foundation models trained on only 5% of the pre-training data achieving up to 6.2% higher mIoU than those trained on the full dataset. Remarkably, OptiDEL demonstrates an average improvement of 4.7% mIoU over competing methods while using 20x less training data.

Figures (6)

• Figure 1: Generate a $\mathcal{V}$-information-rich Vinformation compact dataset to self-supervised pre-train superior medical foundation models. (a) By optimizing $\mathcal{V}$-information, we can generate a smaller unlabeled dataset from a larger unlabeled dataset, with slight differences in the pre-training results of foundation models. (b) Selecting the hardest samples (that are more challenging for foundation models to classify or predict correctly) can optimize the $\mathcal{V}$-information, thereby enhancing model performance. (c) Synthesizing more diverse samples can optimize the $\mathcal{V}$-information.
• Figure 2: The framework of our OptiDEL method to optimize the $\mathcal{V}$-information. The optimization process focuses on two key components: reducing $H_{\mathcal{V}}(Y|X)$ and enhancing $H_{\mathcal{V}}(Y|\varnothing)$. By leveraging a foundation model, we identify and select challenging patches, which helps reduce $H_{\mathcal{V}}(Y|X)$. We then combine every four of these challenging patches to create new images, thereby enhancing $H_{\mathcal{V}}(Y|\varnothing)$.
• Figure 3: Toy example illustrating the impact of hard sample selection on downstream task alignment: the target vector $\boldsymbol{\phi_I}$ represents the downstream task’s objective, the probe vector $\boldsymbol{\phi_D}$ is pre-trained on full Gaussian-distributed data and imperfectly aligns with $\boldsymbol{\phi_I}$ (with the angle $\theta$ quantifying the mismatch), and hard samples are selected near $\boldsymbol{\phi_D}$’s decision boundary, on which a new predictor $\boldsymbol{\phi_S}$ is trained from scratch.
• Figure 4: The numerical experiments of $\mathcal{V}$-information aim to verify the impact of selecting hard samples on the performance of pre-trained foundation models in downstream tasks. Specifically, the correlation between the selection ratio $f$ and the fitting error is examined across different total data-to-parameter ratio $\alpha_t$ and overlap $\theta$. Moreover, as the angle $\theta$ increases, the model's fitting error increases as well. This indicates that using pre-trained models with domain distributions more closely aligned to the target task yields better downstream performance.
• Figure 5: The numerical experiments aim to verify the relationship between fitting error $e$ and pruning ratio $f$ under different difficulty-based sample methods of pre-training foundation models in downstream tasks. The solid line represents the selection of hard samples, while the dotted lines correspond to random selection, easy sample selection, and moderate sample selection in the three plots, respectively. It can be observed that when the total data size is large, selecting difficult samples generally yields better performance.

Theorems & Definitions (5)

• Definition 1
• Proposition 1
• Definition 2
• Definition 3
• Definition 4