Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis

TL;DR

The paper interrogates whether medical specialization of vision-language representations yields truly discriminative lesion-specific features. By visualizing feature distributions with UMAP and probing with linear discriminators across eight modalities, the study finds that improvements to the text encoder, such as using LLMs, often surpass medical pre-training of image encoders in driving discriminative power. It reveals pervasive background biases that can mislead classifiers and notes that publication-derived data (e.g., PMC-15M) can degrade fine-grained feature extraction in certain tasks. The findings suggest prioritizing context-rich text representations and high-quality medical data, while also calling for exploration of text features, 3D/WSI analyses, and local feature representations to fully harness medical VLMs.

Abstract

This study investigates the feature representations produced by publicly available open source medical vision-language models (VLMs). While medical VLMs are expected to capture diagnostically relevant features, their learned representations remain underexplored, and standard evaluations like classification accuracy do not fully reveal if they acquire truly discriminative, lesion-specific features. Understanding these representations is crucial for revealing medical image structures and improving downstream tasks in medical image analysis. This study aims to investigate the feature distributions learned by medical VLMs and evaluate the impact of medical specialization. We analyze the feature distribution of multiple image modalities extracted by some representative medical VLMs across lesion classification datasets on multiple modalities. These distributions were compared them with non-medical VLMs to assess the domain-specific medical training. Our experiments showed that medical VLMs can extract discriminative features that are effective for medical classification tasks. Moreover, it was found that non-medical VLMs with recent improvement with contextual enrichment such as LLM2CLIP produce more refined feature representations. Our results imply that enhancing text encoder is more crucial than training intensively on medical images when developing medical VLMs. Notably, non-medical models are particularly vulnerable to biases introduced by overlaied text strings on images. These findings underscore the need for careful consideration on model selection according to downstream tasks besides potential risks in inference due to background biases such as textual information in images.

Figures (4)

• Figure 1: A exsample of feature distribution on Brain MR Bhuvaji20 dataset and corresponding RISE saliency maps. (a) Feature distribution and example images from the Brain MR Bhuvaji20, generated via LLM2CLIP Huang24. Three clusters are formed according to the anatomical plane, with the central cluster primarily composed of images without tumors. (b) A class activation map from the LLM2CLIP and linear SVM classifiers. The model focus on the lesion area and the orbital region.
• Figure 2: A example of feature distribution on the breast US AlDhabyani20 Dataset and corresponding RISE saliency maps. (a) Feature distribution and example images from the Breast Ultrasound Images Dataset AlDhabyani20, generated via LLaVA-Med. The upper-left and lower-left images are labeled as 'BREAST' and 'AXILLA,' respectively. The tumours in the lower-right images are annotated with lines. (b) Class activation map from the LLM2CLIP and linear SVM classifiers. The model focus on the textual bias written on images.
• Figure 3: Examples of feature distributions for LLaVA-Med, its non-medical counterpart LLaVA, and LLM2CLIP employing an LLM as its text encoder. The upper row shows distributions in Brain MR Bhuvaji20, while the lower row shows distributions in Breast US AlDhabyani20. No significant differences are observed in the feature distributions between LLaVA-Med and other non-medical VLMs employing an LLM as their text encoder.
• Figure 4: Feature distributions in microbe subclasses on HiCervix Cai24 by LLMs trained on PMC-15M Zhang23 and other VLMs. The three on the left (BiomedCLIP Zhang23, LLaVA-Med Li23, LLaVA-Med++ Xie24) are feature distributions from modality-agnostic VLMs trained on PMC-15M, which can be assumed to include cytological images. The two on the upper right (CLIP Radford21, LLaVA Liu23Liu24) are non-medical VLMs. The CXR-CLIP you23 and FLAIR silva2025foundation on the lower right are VLMs specialised for modalities other than cytology, and cytological images were not included in their training data. The model trained on PMC-15M shows no disentangled cluster within the subclasses. Conversely, the VLM on the right, which did not learn cervical images, exhibits a clear cluster structure for each subclass.