Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Medical Scene Reconstruction and Segmentation based on 3D Gaussian Representation

Bin Liu, Wenyan Tian, Huangxin Fu, Zizheng Li, Zhifen He, Bo Li

TL;DR

The paper tackles the challenge of accurate 3D reconstruction and semantic segmentation from sparse medical image slices. It introduces a hybrid explicit-implicit representation that combines 3D Gaussian primitives with a tri-plane global feature field, enabling continuous 3D modeling and joint geometry, appearance, and semantic decoding via an implicit decoder and an InnerGS rasterizer. Key contributions include integrating semantic attributes into Gaussian primitives, enforcing cross-slice consistency with a tri-plane constraint, and demonstrating robust performance across fetal brain ultrasound, prostate MRI, and cardiac MRI under sparse data conditions. The approach yields higher PSNR, SSIM, and lower LPIPS compared to prior 3DGS methods, facilitating efficient, anatomically coherent visualization and clinical analysis with potential broad adoption in multimodal medical imaging.

Abstract

3D reconstruction of medical images is a key technology in medical image analysis and clinical diagnosis, providing structural visualization support for disease assessment and surgical planning. Traditional methods are computationally expensive and prone to structural discontinuities and loss of detail in sparse slices, making it difficult to meet clinical accuracy requirements.To address these challenges, we propose an efficient 3D reconstruction method based on 3D Gaussian and tri-plane representations. This method not only maintains the advantages of Gaussian representation in efficient rendering and geometric representation but also significantly enhances structural continuity and semantic consistency under sparse slicing conditions. Experimental results on multimodal medical datasets such as US and MRI show that our proposed method can generate high-quality, anatomically coherent, and semantically stable medical images under sparse data conditions, while significantly improving reconstruction efficiency. This provides an efficient and reliable new approach for 3D visualization and clinical analysis of medical images.

Medical Scene Reconstruction and Segmentation based on 3D Gaussian Representation

TL;DR

The paper tackles the challenge of accurate 3D reconstruction and semantic segmentation from sparse medical image slices. It introduces a hybrid explicit-implicit representation that combines 3D Gaussian primitives with a tri-plane global feature field, enabling continuous 3D modeling and joint geometry, appearance, and semantic decoding via an implicit decoder and an InnerGS rasterizer. Key contributions include integrating semantic attributes into Gaussian primitives, enforcing cross-slice consistency with a tri-plane constraint, and demonstrating robust performance across fetal brain ultrasound, prostate MRI, and cardiac MRI under sparse data conditions. The approach yields higher PSNR, SSIM, and lower LPIPS compared to prior 3DGS methods, facilitating efficient, anatomically coherent visualization and clinical analysis with potential broad adoption in multimodal medical imaging.

Abstract

3D reconstruction of medical images is a key technology in medical image analysis and clinical diagnosis, providing structural visualization support for disease assessment and surgical planning. Traditional methods are computationally expensive and prone to structural discontinuities and loss of detail in sparse slices, making it difficult to meet clinical accuracy requirements.To address these challenges, we propose an efficient 3D reconstruction method based on 3D Gaussian and tri-plane representations. This method not only maintains the advantages of Gaussian representation in efficient rendering and geometric representation but also significantly enhances structural continuity and semantic consistency under sparse slicing conditions. Experimental results on multimodal medical datasets such as US and MRI show that our proposed method can generate high-quality, anatomically coherent, and semantically stable medical images under sparse data conditions, while significantly improving reconstruction efficiency. This provides an efficient and reliable new approach for 3D visualization and clinical analysis of medical images.
Paper Structure (9 sections, 2 equations, 3 figures, 3 tables)

This paper contains 9 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Three-dimensional medical image reconstruction
  4. Three-dimensional Gaussian splattering and its medical applications
  5. Main Method
  6. Experiments
  7. Sparse slice reconstruction
  8. Semantic segmentation
  9. Conclusion

Figures (3)

  • Figure 1: The results of two groups of fetal brain reconstructions from new perspectives. The left image is the ground truth, and the right image is the reconstructed image.
  • Figure 2: The reconstruction results of the prostate dataset on three axes, including images and semantics. The left image is the ground truth, and the right image is the reconstructed image.
  • Figure 3: The reconstruction results of cardiac MRI images and semantics on three axes. The left image is the ground truth, and the right image is the reconstructed image.