Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction

Yifan Liu, Chenxin Li, Chen Yang, Yixuan Yuan

TL;DR

This work tackles the challenge of real-time reconstruction of deformable endoscopic scenes, where prior NeRF-like approaches are too slow for intraoperative use. It introduces EndoGaussian, a real-time reconstruction framework based on 3D Gaussian Splatting (3DGS) that employs Holistic Gaussian Initialization (HGI) to rapidly seed dense Gaussians from depth predictions and Spatio-temporal Gaussian Tracking (SGT) to model tissue deformation with a lightweight HexPlane-based encoding voxel and a small deformation decoder. The method achieves state-of-the-art reconstruction quality with real-time rendering at ~195 FPS and ~37.85 PSNR, while requiring only about 2 minutes of training per scene and ~2 GB of GPU memory, representing a 100× speedup over existing methods. This enables practical intraoperative applications in RAMIS, providing surgeons with real-time, geometry-consistent visualizations and paving the way for real-time surgical scene understanding and assistance.

Abstract

Reconstructing deformable tissues from endoscopic videos is essential in many downstream surgical applications. However, existing methods suffer from slow rendering speed, greatly limiting their practical use. In this paper, we introduce EndoGaussian, a real-time endoscopic scene reconstruction framework built on 3D Gaussian Splatting (3DGS). By integrating the efficient Gaussian representation and highly-optimized rendering engine, our framework significantly boosts the rendering speed to a real-time level. To adapt 3DGS for endoscopic scenes, we propose two strategies, Holistic Gaussian Initialization (HGI) and Spatio-temporal Gaussian Tracking (SGT), to handle the non-trivial Gaussian initialization and tissue deformation problems, respectively. In HGI, we leverage recent depth estimation models to predict depth maps of input binocular/monocular image sequences, based on which pixels are re-projected and combined for holistic initialization. In SPT, we propose to model surface dynamics using a deformation field, which is composed of an efficient encoding voxel and a lightweight deformation decoder, allowing for Gaussian tracking with minor training and rendering burden. Experiments on public datasets demonstrate our efficacy against prior SOTAs in many aspects, including better rendering speed (195 FPS real-time, 100$\times$ gain), better rendering quality (37.848 PSNR), and less training overhead (within 2 min/scene), showing significant promise for intraoperative surgery applications. Code is available at: \url{https://yifliu3.github.io/EndoGaussian/}.

EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction

TL;DR

This work tackles the challenge of real-time reconstruction of deformable endoscopic scenes, where prior NeRF-like approaches are too slow for intraoperative use. It introduces EndoGaussian, a real-time reconstruction framework based on 3D Gaussian Splatting (3DGS) that employs Holistic Gaussian Initialization (HGI) to rapidly seed dense Gaussians from depth predictions and Spatio-temporal Gaussian Tracking (SGT) to model tissue deformation with a lightweight HexPlane-based encoding voxel and a small deformation decoder. The method achieves state-of-the-art reconstruction quality with real-time rendering at ~195 FPS and ~37.85 PSNR, while requiring only about 2 minutes of training per scene and ~2 GB of GPU memory, representing a 100× speedup over existing methods. This enables practical intraoperative applications in RAMIS, providing surgeons with real-time, geometry-consistent visualizations and paving the way for real-time surgical scene understanding and assistance.

Abstract

Reconstructing deformable tissues from endoscopic videos is essential in many downstream surgical applications. However, existing methods suffer from slow rendering speed, greatly limiting their practical use. In this paper, we introduce EndoGaussian, a real-time endoscopic scene reconstruction framework built on 3D Gaussian Splatting (3DGS). By integrating the efficient Gaussian representation and highly-optimized rendering engine, our framework significantly boosts the rendering speed to a real-time level. To adapt 3DGS for endoscopic scenes, we propose two strategies, Holistic Gaussian Initialization (HGI) and Spatio-temporal Gaussian Tracking (SGT), to handle the non-trivial Gaussian initialization and tissue deformation problems, respectively. In HGI, we leverage recent depth estimation models to predict depth maps of input binocular/monocular image sequences, based on which pixels are re-projected and combined for holistic initialization. In SPT, we propose to model surface dynamics using a deformation field, which is composed of an efficient encoding voxel and a lightweight deformation decoder, allowing for Gaussian tracking with minor training and rendering burden. Experiments on public datasets demonstrate our efficacy against prior SOTAs in many aspects, including better rendering speed (195 FPS real-time, 100 gain), better rendering quality (37.848 PSNR), and less training overhead (within 2 min/scene), showing significant promise for intraoperative surgery applications. Code is available at: \url{https://yifliu3.github.io/EndoGaussian/}.
Paper Structure (11 sections, 8 equations, 3 figures, 2 tables)

This paper contains 11 sections, 8 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries of 3DGS
  4. Holistic Gaussian Initialization
  5. Spatio-temporal Gaussian Tracking
  6. Optimization
  7. Experiments
  8. Experiment settings
  9. Main results
  10. Ablation Study
  11. Conclusion

Figures (3)

  • Figure 1: Illustration of (a) the pipeline of our EndoGaussian framework and (b) benchmarked results of ours against previous SOTAs on ENDONERF dataset wang2022neural.
  • Figure 2: Illustration of the proposed EndoGaussian framework, including a) Holistic Gaussian Initialization, b) Spatio-temporal Gaussian Tracking, and c) Optimization.
  • Figure 3: Illustration of the rendered images of previous works and ours.