SwinTF3D: A Lightweight Multimodal Fusion Approach for Text-Guided 3D Medical Image Segmentation

TL;DR

SwinTF3D tackles the need for adaptable 3D medical segmentation by unifying a fine-tuned visual backbone with a compact, frozen language model to enable text-guided segmentation. The framework fuses text embeddings into segmentation logits through a lightweight biasing mechanism and optional spatial priors, avoiding heavy multimodal decoders. Experiments on BTCV show competitive accuracy with efficient operation, while cross-dataset results on Synapse demonstrate generalization, and prompt-based inference illustrates interactive, semantically guided segmentation. This work offers a practical, interpretable approach to language-conditioned 3D medical imaging that can scale to resource-constrained clinical settings.

Abstract

The recent integration of artificial intelligence into medical imaging has driven remarkable advances in automated organ segmentation. However, most existing 3D segmentation frameworks rely exclusively on visual learning from large annotated datasets restricting their adaptability to new domains and clinical tasks. The lack of semantic understanding in these models makes them ineffective in addressing flexible, user-defined segmentation objectives. To overcome these limitations, we propose SwinTF3D, a lightweight multimodal fusion approach that unifies visual and linguistic representations for text-guided 3D medical image segmentation. The model employs a transformer-based visual encoder to extract volumetric features and integrates them with a compact text encoder via an efficient fusion mechanism. This design allows the system to understand natural-language prompts and correctly align semantic cues with their corresponding spatial structures in medical volumes, while producing accurate, context-aware segmentation results with low computational overhead. Extensive experiments on the BTCV dataset demonstrate that SwinTF3D achieves competitive Dice and IoU scores across multiple organs, despite its compact architecture. The model generalizes well to unseen data and offers significant efficiency gains compared to conventional transformer-based segmentation networks. Bridging visual perception with linguistic understanding, SwinTF3D establishes a practical and interpretable paradigm for interactive, text-driven 3D medical image segmentation, opening perspectives for more adaptive and resource-efficient solutions in clinical imaging.

Figures (9)

• Figure 1: Overall architecture of the proposed SwinTF3D model showing the image and text processing pathways, the fusion module, and the final segmentation output.
• Figure 2: 3D visualization of abdominal organ segmentation results on the BTCV dataset. The reconstructed surfaces are obtained from the predicted segmentation masks generated by SwinTF3D, highlighting accurate anatomical delineation and spatial consistency across multiple organs.
• Figure 3: Architecture of the SwinUNETR-RH model used as the visual encoder. The network employs hierarchical Swin Transformer blocks for volumetric feature extraction and incorporates a lightweight refinement head (RH) for improved boundary precision in organ segmentation.
• Figure 4: Axial view comparison of abdominal organ segmentation on the BTCV dataset. From left to right: the original CT slice, the ground truth segmentation, and the prediction generated by the proposed SwinTF3D framework.
• Figure 5: Average Dice Similarity Coefficient (DSC) per organ obtained using the proposed SwinTF3D framework on the BTCV dataset. The figure highlights the model’s consistent performance across multiple organs, with higher DSC values indicating more accurate segmentations.