Registering Neural 4D Gaussians for Endoscopic Surgery

TL;DR

This work tackles dynamic surgical scene registration using implicit neural representations by leveraging a $4D$ Gaussian Splatting backbone coupled with Spatially Weighted Cluttering to extract robust, spatially-aware features. A deformable scene registration cascade fuses static and deformed Gaussian features to estimate a $6$-DoF transform efficiently, achieving real-time performance. Compared to baselines like DReg-NeRF, the proposed method shows superior rotation/translation accuracy and substantially faster registration, validated on endoscopic StereoMIS data. The approach holds promise for real-time surgical planning, navigation, and robot-assisted training by enabling accurate, deformable registrations in dynamic intraoperative scenes.

Abstract

The recent advance in neural rendering has enabled the ability to reconstruct high-quality 4D scenes using neural networks. Although 4D neural reconstruction is popular, registration for such representations remains a challenging task, especially for dynamic scene registration in surgical planning and simulation. In this paper, we propose a novel strategy for dynamic surgical neural scene registration. We first utilize 4D Gaussian Splatting to represent the surgical scene and capture both static and dynamic scenes effectively. Then, a spatial aware feature aggregation method, Spatially Weight Cluttering (SWC) is proposed to accurately align the feature between surgical scenes, enabling precise and realistic surgical simulations. Lastly, we present a novel strategy of deformable scene registration to register two dynamic scenes. By incorporating both spatial and temporal information for correspondence matching, our approach achieves superior performance compared to existing registration methods for implicit neural representation. The proposed method has the potential to improve surgical planning and training, ultimately leading to better patient outcomes.

Figures (2)

• Figure 1: Overview of the 4D Neural Representation Registration. With the 2D images of the preoperative scene and intraoperative scene, we train two separate neural networks as implicit 4D scene representations. To establish a full scene for the dynamic surgical reconstruction, we proposed a novel registration method for the implicit 4D representations.
• Figure 2: Illustration of the proposed method. We first train two 4D neural representations ($\mathcal{G}_1, \mathcal{G}^\prime_1;\mathcal{G}_2, \mathcal{G}^\prime_2$) with images from a preoperative scene 1 and an intraoperative scene 2, then the implicit features $X_1, X_1^\prime; X_2, X_2^\prime$ of 4D Gaussians is passed into our spatially weighted clustering and feature cascading pipeline. Finally, we register the cascaded output $X^*_1, X^*_2$ for the transformation $\mathbf{\hat{T}}$.