Language-guided Scale-aware MedSegmentor for Lesion Segmentation in Medical Imaging

TL;DR

This work defines Referring Lesion Segmentation (RLS), a task to segment a target lesion in a medical image using a natural language expression, and provides the RefHL-Seg dataset as a dedicated benchmark. It proposes Language-guided Scale-aware MedSegmentor (LSMS), which fuses scale-aware visual knowledge with linguistic guidance via Scale-aware Vision-Language Attention (SVLA) and a Full-Scale Decoder (FSD) to globally model multi-scale multi-modal features. The model is trained with a joint loss combining segmentation and vision-language contrastive terms, and it demonstrates superior performance on RLS and traditional lesion segmentation benchmarks with lower computational costs. The work also shows that textual guidance enhances segmentation quality and provides a strong baseline for future language-guided medical imaging tasks, with code and data to be released.

Abstract

In clinical practice, segmenting specific lesions based on the needs of physicians can significantly enhance diagnostic accuracy and treatment efficiency. However, conventional lesion segmentation models lack the flexibility to distinguish lesions according to specific requirements. Given the practical advantages of using text as guidance, we propose a novel model, Language-guided Scale-aware MedSegmentor (LSMS), which segments target lesions in medical images based on given textual expressions. We define this as a new task termed Referring Lesion Segmentation (RLS). To address the lack of suitable benchmarks for RLS, we construct a vision-language medical dataset named Reference Hepatic Lesion Segmentation (RefHL-Seg). LSMS incorporates two key designs: (i) Scale-Aware Vision-Language attention module, which performs visual feature extraction and vision-language alignment in parallel. By leveraging diverse convolutional kernels, this module acquires rich visual representations and interacts closely with linguistic features, thereby enhancing the model's capacity for precise object localization. (ii) Full-Scale Decoder, which globally models multi-modal features across multiple scales and captures complementary information between them to accurately delineate lesion boundaries. Additionally, we design a specialized loss function comprising both segmentation loss and vision-language contrastive loss to better optimize cross-modal learning. We validate the performance of LSMS on RLS as well as on conventional lesion segmentation tasks across multiple datasets. Our LSMS consistently achieves superior performance with significantly lower computational cost. Code and datasets will be released.

Figures (6)

• Figure 1: Comparison between Referring Lesion Segmentation and conventional lesion segmentation tasks. In the images displaying segmentation results, the regions highlighted in red represent the Ground Truth. For intuitive correspondence with the left-right references in the text, all CT images in this paper have been mirrored horizontally.
• Figure 2: Comparison of Transformer-based architectures. (a) Existing architectures for related tasks. (b) our LSMS for RLS.
• Figure 3: Qualitative results of different approaches. The red regions denote the Ground Truth, while the green regions represent the segmentation results of our LSMS, LSMS (w/o FSD), LViT li2023lvit and LAVT yang2022lavt. In sample (b), for ease of observation, the key region within the image have been enlarged, with a yellow rectangular box serving as a reference for location.
• Figure 4: (a) An illustration of LSMS. Initially, the input image and the reference expression are embedded separately through the vision embedding block and the BERT devlin2018bert language encoder, yielding visual feature $V_1$ and linguistic feature $L$, which are then fed into the language-guided vision encoder. The encoder incorporates the Scale-aware Vision-Language Attention (SVLA) module to interact between visual knowledge from different receptive fields and linguistic features. The encoder blocks at each stage learn rich multi-modal features $F_i, i \in \{ 1,2,3,4\}$, which are subsequently fed into the full-scale decoder. Through the Position Alignment block, $F_i, i \in \{ 1,2,3,4\}$ uniformly resize the feature maps of various scales while preserving channel disparities, resulting in $P_i, i \in \{ 1,2,3,4\}$. Features $P_i, i \in \{ 2,3,4\}$ are then globally modeled across scales for final segmentation. (b) An illustration of the operational mechanism of the vision-language contrastive loss $\mathcal{L}_{Con}$.
• Figure 5: (a) An illustration of the encoder block in the Language-Guided Vision Encoder. (b) An illustration of the Scale-aware Vision-Language Attention module.