Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian Pancakes: Geometrically-Regularized 3D Gaussian Splatting for Realistic Endoscopic Reconstruction

Sierra Bonilla, Shuai Zhang, Dimitrios Psychogyios, Danail Stoyanov, Francisco Vasconcelos, Sophia Bano

TL;DR

Gaussian Pancakes addresses the challenge of reconstructing photorealistic and geometrically accurate 3D colon surfaces from monocular endoscopy by fusing 3D Gaussian Splatting with RNNSLAM. It augments the Gaussian representation with depth regularization and a geometric pancaking constraint to align Gaussians with surface normals, enabling robust novel-view synthesis and explicit geometry. The approach yields substantial gains in perceptual quality (e.g., PSNR up to 40.34 and SSIM up to 0.97) and dramatic speedups (>100 FPS rendering, training times ≈2% of NeRF-based methods), which support near real-time clinical use. Performance depends on the quality of RNNSLAM’s pose/depth estimates, and future work will focus on stronger point-cloud generation and depth accuracy to broaden applicability in diverse endoscopic environments.

Abstract

Within colorectal cancer diagnostics, conventional colonoscopy techniques face critical limitations, including a limited field of view and a lack of depth information, which can impede the detection of precancerous lesions. Current methods struggle to provide comprehensive and accurate 3D reconstructions of the colonic surface which can help minimize the missing regions and reinspection for pre-cancerous polyps. Addressing this, we introduce 'Gaussian Pancakes', a method that leverages 3D Gaussian Splatting (3D GS) combined with a Recurrent Neural Network-based Simultaneous Localization and Mapping (RNNSLAM) system. By introducing geometric and depth regularization into the 3D GS framework, our approach ensures more accurate alignment of Gaussians with the colon surface, resulting in smoother 3D reconstructions with novel viewing of detailed textures and structures. Evaluations across three diverse datasets show that Gaussian Pancakes enhances novel view synthesis quality, surpassing current leading methods with a 18% boost in PSNR and a 16% improvement in SSIM. It also delivers over 100X faster rendering and more than 10X shorter training times, making it a practical tool for real-time applications. Hence, this holds promise for achieving clinical translation for better detection and diagnosis of colorectal cancer.

Gaussian Pancakes: Geometrically-Regularized 3D Gaussian Splatting for Realistic Endoscopic Reconstruction

TL;DR

Gaussian Pancakes addresses the challenge of reconstructing photorealistic and geometrically accurate 3D colon surfaces from monocular endoscopy by fusing 3D Gaussian Splatting with RNNSLAM. It augments the Gaussian representation with depth regularization and a geometric pancaking constraint to align Gaussians with surface normals, enabling robust novel-view synthesis and explicit geometry. The approach yields substantial gains in perceptual quality (e.g., PSNR up to 40.34 and SSIM up to 0.97) and dramatic speedups (>100 FPS rendering, training times ≈2% of NeRF-based methods), which support near real-time clinical use. Performance depends on the quality of RNNSLAM’s pose/depth estimates, and future work will focus on stronger point-cloud generation and depth accuracy to broaden applicability in diverse endoscopic environments.

Abstract

Within colorectal cancer diagnostics, conventional colonoscopy techniques face critical limitations, including a limited field of view and a lack of depth information, which can impede the detection of precancerous lesions. Current methods struggle to provide comprehensive and accurate 3D reconstructions of the colonic surface which can help minimize the missing regions and reinspection for pre-cancerous polyps. Addressing this, we introduce 'Gaussian Pancakes', a method that leverages 3D Gaussian Splatting (3D GS) combined with a Recurrent Neural Network-based Simultaneous Localization and Mapping (RNNSLAM) system. By introducing geometric and depth regularization into the 3D GS framework, our approach ensures more accurate alignment of Gaussians with the colon surface, resulting in smoother 3D reconstructions with novel viewing of detailed textures and structures. Evaluations across three diverse datasets show that Gaussian Pancakes enhances novel view synthesis quality, surpassing current leading methods with a 18% boost in PSNR and a 16% improvement in SSIM. It also delivers over 100X faster rendering and more than 10X shorter training times, making it a practical tool for real-time applications. Hence, this holds promise for achieving clinical translation for better detection and diagnosis of colorectal cancer.
Paper Structure (10 sections, 6 equations, 5 figures, 3 tables)

This paper contains 10 sections, 6 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries of RNNSLAM
  4. Preliminaries 3D Gaussian Splatting
  5. Proposed Gaussian Pancakes: Extending 3D GS for Endoscopy
  6. Experiments
  7. Dataset and Evaluation Metrics:
  8. Results and Discussion
  9. Conclusion
  10. Appendix

Figures (5)

  • Figure 1: Illustrating the benefit of Gaussian Pancakes, arrows indicating Gaussians' principal normal direction, A) Gaussian Splatting without Pancaking (PSNR = 37.54) and B) Gaussian Splatting with Pancaking (PSNR = 38.50), on a synthetic video sequence generated by a colonoscopy simulator zhang2020template; C) Sparse point cloud from SfM and D) point cloud from RNNSLAM on the In-Vivo dataset ma2021rnnslam.
  • Figure 2: Proposed Gaussian Pancakes' pipeline highlighting our contributions: A) RNNSLAM for mesh, camera poses, & depth maps; B) 3D GS initialization, C) Geometric & depth regularizations, distinguishing our approach from traditional 3D GS.
  • Figure A.1: Test images from the In-Vivo dataset showcasing the artifacts that arise in other methods.
  • Figure A.2: Image showing the surface reconstruction from the basic 3D GS method (top) and from Gaussian Pancakes (bottom).
  • Figure A.3: Depth renderings showing effect of systematically adding all changes to the basic 3D GS method.