BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation

TL;DR

BiPVL-Seg addresses the gap in medical image segmentation by coupling bidirectional progressive vision-language fusion with global-local embedding alignment in an end-to-end framework. It introduces BiFusion blocks to exchange information at each encoder stage and a hierarchical contrastive objective that aligns class-level descriptions and fine-grained concepts with visual features. The approach yields state-of-the-art performance across CT and MR datasets (AMOS22, MM-WHS, MSD-Brain) and across multi-class segmentation tasks, while validating the contribution of both architectural and training innovations. This work paves the way for clinically leverageable multimodal segmentation that integrates textual clinical knowledge directly into the segmentation pipeline.

Abstract

Medical image segmentation typically relies solely on visual data, overlooking the rich textual information clinicians use for diagnosis. Vision-language models attempt to bridge this gap, but existing approaches often process visual and textual features independently, resulting in weak cross-modal alignment. Simple fusion techniques fail due to the inherent differences between spatial visual features and sequential text embeddings. Additionally, medical terminology deviates from general language, limiting the effectiveness of off-the-shelf text encoders and further hindering vision-language alignment. We propose BiPVL-Seg, an end-to-end framework that integrates vision-language fusion and embedding alignment through architectural and training innovations, where both components reinforce each other to enhance medical image segmentation. BiPVL-Seg introduces bidirectional progressive fusion in the architecture, which facilitates stage-wise information exchange between vision and text encoders. Additionally, it incorporates global-local contrastive alignment, a training objective that enhances the text encoder's comprehension by aligning text and vision embeddings at both class and concept levels. Extensive experiments on diverse medical imaging benchmarks across CT and MR modalities demonstrate BiPVL-Seg's superior performance when compared with state-of-the-art methods in complex multi-class segmentation. Source code is available in this GitHub repository.

Figures (5)

• Figure 1: (a) Comparison of our bidirectional progressive fusion (iv) with prior VLM architectures (i–iii), showing improved cross-modal interaction. (b) t-SNE visualization shows an improved concept separation in BiPVL-Seg’s text encoder (right) compared to ClinicalBERT (left), demonstrating the benefit of global-local alignment.
• Figure 2: (a) BiPVL-Seg, an end-to-end training pipeline with segmentation, class embedding, and global-local alignment losses. (b) Bidirectional progressive fusion between encoders via BiFusion blocks at each stage. (c) $\mathcal{L}_{ClassEmbed}$: Mapping encoder embeddings to class-specific embeddings with decoder supervision. (d) $\mathcal{L}_{alignment}$: Balanced global-local alignment linking class-level text embeddings to visual features, while fine-grained concept-level alignment uses hard negatives.
• Figure 3: A demo of curating concept descriptions of each class in our datasets using ChatGPT.
• Figure 4: Qualitative visualizations of randomly selected image slices from AMOS22, MM-WHS (CT), and MM-WHS (MR) and MSD-Brain (MR). Comparison of BiPVL-Seg against other models, with red boxes indicating areas where BiPVL-Seg outperforms.
• Figure 5: Text embeddings are created from concept embeddings. Different shades of concept embeddings emphasize that they have different weights.