Spatial-aware Symmetric Alignment for Text-guided Medical Image Segmentation

TL;DR

This work tackles the semantic gap in text-guided medical image segmentation by addressing the joint interpretation of diagnostic and descriptive texts and the need for explicit spatial constraints. It introduces Spatial-aware Symmetric Alignment (SSA), which combines Dual-granularity Symmetric Optimal Transport (DSOT) alignment with Composite Directional Guidance (CDG) to enforce region-text correspondences and spatial priors. DSOT integrates a global InfoNCE-based alignment with a fine-grained, symmetric optimal transport on bi-aggregated features, using a cost matrix $M \in \mathbb{R}^{N \times L}$ where $M_{ij} = 1 - \cos(\mathbf{f}_{\text{img},i}, \mathbf{f}_{\text{txt},j})$, solved via Sinkhorn with entropy $\epsilon = 3\times 10^{-2}$; CDG builds region-level masks from directional cues through $M_{\text{guide}} = A_{\text{norm}} \odot M_{\text{pri}}$ and enforces a consistency loss $\mathcal{L}_{\text{guide}}$. Experiments on QaTa-COV19 and MosMedData+ demonstrate SOTA Dice scores and show data-efficient learning, highlighting SSA's capacity to segment lesions under complex spatial expressions and improve clinical applicability. The findings suggest that joint global-local cross-modal alignment and explicit spatial supervision can significantly enhance text-guided medical image segmentation in real-world settings.

Abstract

Text-guided Medical Image Segmentation has shown considerable promise for medical image segmentation, with rich clinical text serving as an effective supplement for scarce data. However, current methods have two key bottlenecks. On one hand, they struggle to process diagnostic and descriptive texts simultaneously, making it difficult to identify lesions and establish associations with image regions. On the other hand, existing approaches focus on lesions description and fail to capture positional constraints, leading to critical deviations. Specifically, with the text "in the left lower lung", the segmentation results may incorrectly cover both sides of the lung. To address the limitations, we propose the Spatial-aware Symmetric Alignment (SSA) framework to enhance the capacity of referring hybrid medical texts consisting of locational, descriptive, and diagnostic information. Specifically, we propose symmetric optimal transport alignment mechanism to strengthen the associations between image regions and multiple relevant expressions, which establishes bi-directional fine-grained multimodal correspondences. In addition, we devise a composite directional guidance strategy that explicitly introduces spatial constraints in the text by constructing region-level guidance masks. Extensive experiments on public benchmarks demonstrate that SSA achieves state-of-the-art (SOTA) performance, particularly in accurately segmenting lesions characterized by spatial relational constraints.

Figures (4)

• Figure 1: Existing multi-modal methods perform alignment by adopting either (a) Global Alignment or (b) Local Alignment based on uni-aggregation. In this paper, we propose (c) DSOT alignment, which performs both global and local alignment on bi-aggregated features, implemented via InfoNCE and symmetric OT algorithms respectively.
• Figure 2: The overall architecture of our proposed SSA framework. Built upon a U-Net-based baseline, our approach introduces two modules. The DSOT module reinforces the alignment between visual regions and relevant textual descriptions. And the CDG module focuses on accurately segmenting lesions characterized by spatial relational constraints.
• Figure 3: Qualitative results on the QaTa-COV19 dataset. Our method achieves better overlap with the Ground Truth (GT) compared to other methods. Colors represent True Positive (TP), False Negative (FN), and False Positive (FP) regions in orange, green, and yellow, respectively.
• Figure 4: Visualization of the Composite Directional Guidance. The raw attention map covers regions beyond the “lower left lung", leading to potential critical deviations. Our approach generates a region-level prior mask from the text and further introduces a guided mask to enhance consistency with spatial textual descriptions.