Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Refining 3D Medical Segmentation with Verbal Instruction

Kangxian Xie, Jiancheng Yang, Nandor Pinter, Chao Wu, Behzad Bozorgtabar, Mingchen Gao

Abstract

Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.

Refining 3D Medical Segmentation with Verbal Instruction

Abstract

Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.
Paper Structure (14 sections, 4 equations, 7 figures, 1 table)

This paper contains 14 sections, 4 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. CoWTalk Dataset Synthesis
  4. Interactive Shape Refinement Objective
  5. Methods
  6. Shape Encoding via Latent Query Cross-Attention
  7. Instruction Encoding and Multi-modal Interaction
  8. Implicit Point-wise Decoding with CNN Feature
  9. Training and Inference
  10. Experiments
  11. Baseline Comparisons
  12. Multi-modal Input Ablation on Error Categories
  13. Radiologist Instruction on Real Error
  14. Conclusion

Figures (7)

  • Figure 1: The overall workflow of iterative text-based refinement for 3D medical segmentation shapes. The green arrows and 6 steps represent the iterative workflow.
  • Figure 2: Centerline refinement and Segment Selection. A to E shows the steps for refining the centerline representation, while F illustrates the partial selection of a segment.
  • Figure 3: Examples of generated instructions in CoWTalk.
  • Figure 4: Overview of the proposed instruction-guided medical shape refinement pipeline.
  • Figure 5: Qualitative refinement results with shape-only input from CoWTalk, and only 3 corrections are highlighted for each sample. Some correction descriptions are omitted.
  • ...and 2 more figures