Confounder-Aware Medical Data Selection for Fine-Tuning Pretrained Vision Models
Anyang Ji, Qingbo Kang, Wei Xu, Changfan Wang, Kang Li, Qicheng Lao
TL;DR
The paper addresses the problem of efficient fine-tuning of pretrained vision models on medical imaging data under annotation cost and privacy constraints. It proposes a confounder-aware two-stage framework: first, identify confounding variables using pre-trained vision models and 2D feature clustering, then select a coreset via distance-based sampling that preserves the natural data distribution under a fixed budget $N$ per class. Feature extraction uses a Masked Autoencoder (MAE) to produce latent representations, followed by 2D dimensionality reduction and class-wise clustering to define confounder groups. Empirical results on LEPset, NeoJaundice, and PatchCamelyon show state-of-the-art gains over several baselines, with pronounced improvements at small coreset sizes and even full-data parity or supremacy in some cases. The work highlights practical relevance for privacy-preserving, data-efficient medical-imaging fine-tuning, while noting the need to tune clustering parameters and potential extensions to automate this process.
Abstract
The emergence of large-scale pre-trained vision foundation models has greatly advanced the medical imaging field through the pre-training and fine-tuning paradigm. However, selecting appropriate medical data for downstream fine-tuning remains a significant challenge considering its annotation cost, privacy concerns, and the detrimental effects of confounding variables. In this work, we present a confounder-aware medical data selection approach for medical dataset curation aiming to select minimal representative data by strategically mitigating the undesirable impact of confounding variables while preserving the natural distribution of the dataset. Our approach first identifies confounding variables within data and then develops a distance-based data selection strategy for confounder-aware sampling with a constrained budget in the data size. We validate the superiority of our approach through extensive experiments across diverse medical imaging modalities, highlighting its effectiveness in addressing the substantial impact of confounding variables and enhancing the fine-tuning efficiency in the medical imaging domain, compared to other data selection approaches.