TGC-Net: A Structure-Aware and Semantically-Aligned Framework for Text-Guided Medical Image Segmentation

TL;DR

Text-guided medical image segmentation faces three core gaps when using general vision-language models: preserving fine-grained anatomical structure, capturing domain-specific medical semantics, and calibrating cross-modal alignment. TGC-Net addresses these with a lightweight tri-gap calibration: the Semantic–Structural Synergy Encoder (SSE) fuses CLIP ViT with a CNN to retain structure, the Domain–Augmented Text Encoder (DATE) injects medical knowledge from large language models, and the Vision–Language Calibration Module (VLCM) refines cross-modal matches in a unified space. The approach achieves state-of-the-art Dice and mIoU on five chest X-ray and thoracic CT datasets with about 10.3 million trainable parameters, validated by extensive ablations showing each module’s contribution. Overall, TGC-Net demonstrates that carefully targeted, task-specific adaptations of pre-aligned vision-language models can yield precise, robust, and clinically meaningful segmentation results while maintaining high parameter efficiency.

Abstract

Text-guided medical segmentation enhances segmentation accuracy by utilizing clinical reports as auxiliary information. However, existing methods typically rely on unaligned image and text encoders, which necessitate complex interaction modules for multimodal fusion. While CLIP provides a pre-aligned multimodal feature space, its direct application to medical imaging is limited by three main issues: insufficient preservation of fine-grained anatomical structures, inadequate modeling of complex clinical descriptions, and domain-specific semantic misalignment. To tackle these challenges, we propose TGC-Net, a CLIP-based framework focusing on parameter-efficient, task-specific adaptations. Specifically, it incorporates a Semantic-Structural Synergy Encoder (SSE) that augments CLIP's ViT with a CNN branch for multi-scale structural refinement, a Domain-Augmented Text Encoder (DATE) that injects large-language-model-derived medical knowledge, and a Vision-Language Calibration Module (VLCM) that refines cross-modal correspondence in a unified feature space. Experiments on five datasets across chest X-ray and thoracic CT modalities demonstrate that TGC-Net achieves state-of-the-art performance with substantially fewer trainable parameters, including notable Dice gains on challenging benchmarks.

Figures (5)

• Figure 1: TGC-Net: Achieving precise localization via parameter-efficient adaptation. (a) Qualitatively, we visualize the feature similarity maps between text prompts and image patches. Standard and fine-tuned CLIP suffer from severe noise and misalignment (columns 2-3), whereas our method produces a purified map that precisely highlights the infection area. (b) Quantitatively, benefiting from our proposed task-specific modules, TGC-Net achieves state-of-the-art accuracy (mDice) on MosMedData+ with substantially fewer trainable parameters than existing methods.
• Figure 2: Overall architectural comparison. (a) Typical text-guided medical segmentation frameworks rely on heavy dual encoders with stacked cross-modal interaction and U-shaped decoders. (b) The dual-branch fusion architecture employs N layers of cross-attention in the text branch to align visual and textual features. (c) Our proposed TGC-Net builds on pre-aligned CLIP encoders and introduces lightweight SSE, DATE, and VLCM modules, resulting in a more compact yet expressive architecture.
• Figure 3: (a) Overview of the proposed Tri-Gap Calibration Network (TGC-Net). The framework comprises three core components that jointly address the Structural, Semantic, and Alignment gaps in language-guided medical image segmentation. (b) The SSE integrates global semantic features from the CLIP visual encoder with fine-grained structural cues extracted by a lightweight CNN, producing a feature representation enriched with both high-level semantic information and fine-grained structural details. (c) The DATE enhances the primary text prompt with domain-specific medical semantics via cross-attention, generating knowledge-infused textual features. (d) The VLCM refines and re-aligns visual and textual embeddings within the medical domain through gated cross-attention interactions. The calibrated multi-modal features are finally fed into the CAT-Seg decoder to generate the segmentation mask.
• Figure 4: Qualitative comparison on the MosMedData+ and QaTa-COVID19 datasets. The first three rows display results from MosMedData+, while the bottom three rows correspond to QaTa-COVID19.
• Figure 5: Comparison of trainable parameters across different models.