Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models

TL;DR

This study systematically investigates transferring Vision-Language Segmentation Models (VLSMs) to 2D medical image segmentation, evaluating four VLSMs built on CLIP and BiomedCLIP across $11$ datasets with carefully engineered language prompts. The authors design automated prompts from 14 attributes into multiple types to probe how textual cues interact with image representations during finetuning and zero-shot inference. Results show that while finetuned VLSMs can be competitive with state-of-the-art segmentation methods and exhibit robustness to distribution shifts, not all VLSMs effectively utilize language prompts, with image features often driving performance. The work provides an open-source evaluation framework, enriched datasets, and prompts to spur further research into robust, open-vocabulary medical segmentation and prompt-driven interpretability.

Abstract

Medical image segmentation allows quantifying target structure size and shape, aiding in disease diagnosis, prognosis, surgery planning, and comprehension.Building upon recent advancements in foundation Vision-Language Models (VLMs) from natural image-text pairs, several studies have proposed adapting them to Vision-Language Segmentation Models (VLSMs) that allow using language text as an additional input to segmentation models. Introducing auxiliary information via text with human-in-the-loop prompting during inference opens up unique opportunities, such as open vocabulary segmentation and potentially more robust segmentation models against out-of-distribution data. Although transfer learning from natural to medical images has been explored for image-only segmentation models, the joint representation of vision-language in segmentation problems remains underexplored. This study introduces the first systematic study on transferring VLSMs to 2D medical images, using carefully curated $11$ datasets encompassing diverse modalities and insightful language prompts and experiments. Our findings demonstrate that although VLSMs show competitive performance compared to image-only models for segmentation after finetuning in limited medical image datasets, not all VLSMs utilize the additional information from language prompts, with image features playing a dominant role. While VLSMs exhibit enhanced performance in handling pooled datasets with diverse modalities and show potential robustness to domain shifts compared to conventional segmentation models, our results suggest that novel approaches are required to enable VLSMs to leverage the various auxiliary information available through language prompts. The code and datasets are available at https://github.com/naamiinepal/medvlsm.

Figures (6)

• Figure 1: CRIS and CLIPSeg-variants include Text and Image encoders, an Aggregator, and a Vision-Language Decoder.
• Figure 2: Zero-shot and finetuning performance of CRIS, CLIPSeg, BiomedCLIPSeg, and BiomedCLIPSeg-D model on non-radiology (first two rows) and radiology datasets (last row). Finetuning using the prompts improves performance compared to the empty prompt, particularly in multi-class settings.
• Figure 3: Relative change in percentage dice score on replacing attribute values by a random uncommon English word (left of vertical lines) or semantically opposite value such as replacing 'large' with 'small' (right of vertical lines) in prompt P6.
• Figure 4: Examples of images with the highest drops in dice score for two datasets when values for sensitive attributes are replaced with another value within the value set of the attributes in the dataset in P6.
• Figure 5: Visualization of CRIS's performance when prompt attributes are changed using a wrong attribute value. For each medical image, three corresponding masks are displayed: ground truth mask, output mask for the corresponding prompt, and output mask after altering an attribute value of the prompts.