EndoWave: Rational-Wavelet 4D Gaussian Splatting for Endoscopic Reconstruction

TL;DR

EndoWave addresses the challenge of accurate 3D reconstruction in dynamic endoscopic scenes by introducing a unified 4D Gaussian Splatting representation trained directly in the spatio-temporal domain. It integrates a flow-based geometric constraint to ground motion in observed pixel dynamics and employs a multi-resolution rational wavelet loss to preserve both global structure and high-frequency details, particularly under specular highlights. The approach demonstrates state-of-the-art reconstruction quality on EndoNeRF and StereoMIS, achieving high PSNR and SSIM while maintaining interactive rendering speeds (e.g., $\text{FPS} = 86$ on EndoNeRF and $77$ on StereoMIS). By avoiding two-stage canonical/deformation pipelines and leveraging explicit 4D geometry with motion-aware supervision, EndoWave offers a practical path toward real-time, accurate surgical scene reconstruction with potential impact on intraoperative navigation and planning.

Abstract

In robot-assisted minimally invasive surgery, accurate 3D reconstruction from endoscopic video is vital for downstream tasks and improved outcomes. However, endoscopic scenarios present unique challenges, including photometric inconsistencies, non-rigid tissue motion, and view-dependent highlights. Most 3DGS-based methods that rely solely on appearance constraints for optimizing 3DGS are often insufficient in this context, as these dynamic visual artifacts can mislead the optimization process and lead to inaccurate reconstructions. To address these limitations, we present EndoWave, a unified spatio-temporal Gaussian Splatting framework by incorporating an optical flow-based geometric constraint and a multi-resolution rational wavelet supervision. First, we adopt a unified spatio-temporal Gaussian representation that directly optimizes primitives in a 4D domain. Second, we propose a geometric constraint derived from optical flow to enhance temporal coherence and effectively constrain the 3D structure of the scene. Third, we propose a multi-resolution rational orthogonal wavelet as a constraint, which can effectively separate the details of the endoscope and enhance the rendering performance. Extensive evaluations on two real surgical datasets, EndoNeRF and StereoMIS, demonstrate that our method EndoWave achieves state-of-the-art reconstruction quality and visual accuracy compared to the baseline method.

Figures (4)

• Figure 1: Wavelet decomposition visualization on EndoNeRF dataset. We propose a rational wavelet decomposition that is effective for endoscopic scenarios. Our proposed rational wavelet is compared against the standard Orthogonal Haar and Biorthogonal bior6.8, visualizing their respective LL, LH, and HL components.
• Figure 2: Overall of the proposed framework. We take RGB-D frames as input to initialize a set of 4D Gaussian Splatting primitives $\mathcal{G}$. Each primitive is decomposed into a conditional 3D Gaussian for its spatial representation and a marginal 1D Gaussian to model its temporal dynamics. The primitives are then jointly optimized using a composite loss function with RGB, depth, optical flow, and multi-scale wavelet supervision. After training, the model can render high-fidelity, time-evolving color and depth maps from novel viewpoints at any given time $t$.
• Figure 3: Qualitative results for novel view synthesis. Left: Results on the StereoMIS stereomis dataset, with magnified details. Center and Right: Comparison of the Cutting and Pulling sequences from the EndoNeRF endonerf dataset, respectively. The last column of each sequence is the error map, dark purple indicates low error, and yellow indicates high error.
• Figure 4: Qualitative 3D Model comparison on the EndoNeRF dataset