Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques

TL;DR

This work addresses the limitation of single-scale learning in medical vision-language foundation models by introducing a multi-scale framework that jointly leverages global, local, instance, and modality-scale information. It defines four complementary losses—$\mathcal{L}_g$, $\mathcal{L}_l$, $\mathcal{L}_{im}$, and $\mathcal{L}_{mr}$—and a unified objective $\mathcal{L} = \lambda_1 \mathcal{L}_g + \lambda_2 \mathcal{L}_l + \lambda_3 \mathcal{L}_{im} + \lambda_4 \mathcal{L}_{mr}$ to train a medical vision-language foundation model. Across six open-source radiography datasets and four clinical tasks (classification, segmentation, zero-shot classification, and phase grounding), the method outperforms multiple baselines, demonstrating strong cross-scale and cross-modal representation learning. The study highlights that multi-scale integration yields richer, more granular medical representations with potential to enhance automated diagnostic systems and clinical image analysis.

Abstract

The development of medical vision-language foundation models has attracted significant attention in the field of medicine and healthcare due to their promising prospect in various clinical applications. While previous studies have commonly focused on feature learning at a single learning scale, investigation on integrating multi-scale information is lacking, which may hinder the potential for mutual reinforcement among these features. This paper aims to bridge this gap by proposing a method that effectively exploits multi-scale information to enhance the performance of medical foundation models. The proposed method simultaneously exploits features at the local, instance, modality and global aspects, facilitating comprehensive representation learning within the models. We evaluate the effectiveness of the proposed method on six open-source datasets across different clinical tasks, demonstrating its ability to enhance the performance of medical foundation models.

Figures (2)

• Figure 1: The proposed foundation model. Four scales of features are simultaneously exploited to enhance the representation learning of the model.
• Figure 2: Visualization of Phase Grounding. We visualize the correlation between Radiographic Images and Clinical Reports on MS-CXR Dataset. The black bounding box represents the ground truth, while deeper color of red indicates higher degree of similarity.