Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TextDiffSeg: Text-guided Latent Diffusion Model for 3d Medical Images Segmentation

Kangbo Ma

TL;DR

TextDiffSeg tackles the challenge of efficient, accurate 3D medical image segmentation by introducing a text-guided diffusion framework that operates in a 3D latent space. It builds a cross-modal embedding between volumetric images and natural language descriptions through a 3D image encoder, a BioBERT-based text encoder, and a 3D cross-modal attention module, complemented by a 3D label encoder and a conditional denoising module. A joint loss combining segmentation and denoising objectives enables end-to-end training and robust performance across organ and tumor segmentation tasks. Empirical results on kidney, pancreas, liver, and colon datasets demonstrate state-of-the-art accuracy and generalization, with ablations confirming the importance of text fusion, image features, and label encoding. The framework holds promise for clinically interactive workflows and scalable deployment in diverse medical imaging scenarios.

Abstract

Diffusion Probabilistic Models (DPMs) have demonstrated significant potential in 3D medical image segmentation tasks. However, their high computational cost and inability to fully capture global 3D contextual information limit their practical applications. To address these challenges, we propose a novel text-guided diffusion model framework, TextDiffSeg. This method leverages a conditional diffusion framework that integrates 3D volumetric data with natural language descriptions, enabling cross-modal embedding and establishing a shared semantic space between visual and textual modalities. By enhancing the model's ability to recognize complex anatomical structures, TextDiffSeg incorporates innovative label embedding techniques and cross-modal attention mechanisms, effectively reducing computational complexity while preserving global 3D contextual integrity. Experimental results demonstrate that TextDiffSeg consistently outperforms existing methods in segmentation tasks involving kidney and pancreas tumors, as well as multi-organ segmentation scenarios. Ablation studies further validate the effectiveness of key components, highlighting the synergistic interaction between text fusion, image feature extractor, and label encoder. TextDiffSeg provides an efficient and accurate solution for 3D medical image segmentation, showcasing its broad applicability in clinical diagnosis and treatment planning.

TextDiffSeg: Text-guided Latent Diffusion Model for 3d Medical Images Segmentation

TL;DR

TextDiffSeg tackles the challenge of efficient, accurate 3D medical image segmentation by introducing a text-guided diffusion framework that operates in a 3D latent space. It builds a cross-modal embedding between volumetric images and natural language descriptions through a 3D image encoder, a BioBERT-based text encoder, and a 3D cross-modal attention module, complemented by a 3D label encoder and a conditional denoising module. A joint loss combining segmentation and denoising objectives enables end-to-end training and robust performance across organ and tumor segmentation tasks. Empirical results on kidney, pancreas, liver, and colon datasets demonstrate state-of-the-art accuracy and generalization, with ablations confirming the importance of text fusion, image features, and label encoding. The framework holds promise for clinically interactive workflows and scalable deployment in diverse medical imaging scenarios.

Abstract

Diffusion Probabilistic Models (DPMs) have demonstrated significant potential in 3D medical image segmentation tasks. However, their high computational cost and inability to fully capture global 3D contextual information limit their practical applications. To address these challenges, we propose a novel text-guided diffusion model framework, TextDiffSeg. This method leverages a conditional diffusion framework that integrates 3D volumetric data with natural language descriptions, enabling cross-modal embedding and establishing a shared semantic space between visual and textual modalities. By enhancing the model's ability to recognize complex anatomical structures, TextDiffSeg incorporates innovative label embedding techniques and cross-modal attention mechanisms, effectively reducing computational complexity while preserving global 3D contextual integrity. Experimental results demonstrate that TextDiffSeg consistently outperforms existing methods in segmentation tasks involving kidney and pancreas tumors, as well as multi-organ segmentation scenarios. Ablation studies further validate the effectiveness of key components, highlighting the synergistic interaction between text fusion, image feature extractor, and label encoder. TextDiffSeg provides an efficient and accurate solution for 3D medical image segmentation, showcasing its broad applicability in clinical diagnosis and treatment planning.

Paper Structure

This paper contains 16 sections, 6 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview of TextDiffSeg
  4. Cross-modal Embedding
  5. Image Encoder
  6. Text Encoder
  7. 3D Cross-modal Attention
  8. Label Embedding
  9. Conditional Denoising Module
  10. Loss Function
  11. Experiment
  12. Dataset
  13. Implentation Details
  14. Comparison with SOTA Methods
  15. Ablation Study
  16. ...and 1 more sections

Figures (2)

  • Figure 1: The overview of TextDiffSeg consisting training phase and testing phase.
  • Figure 2: Qualitative visualizations of our method and baseline approaches on liver tumor, kidney tumor, pancreas tumor and colon cancer segmentation tasks.