Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction

TL;DR

Free-SurGS addresses the challenge of SfM-free real-time surgical scene reconstruction by jointly optimizing camera poses and a 3D Gaussian representation from monocular videos. It leverages optical-flow priors to guide projection flow and applies a consistency check to filter out unreliable correspondences, enabling robust pose estimation in texture-poor, photometrically noisy scenes. Experiments on the SCARED dataset demonstrate superior novel view synthesis and pose accuracy compared with SfM-free baselines, while maintaining real-time rendering. The approach reduces preprocessing time and improves robustness for endoscopic visualization, with potential impact on augmented reality, surgical planning, and training, though dynamic tissue deformations remain a limitation for future work.

Abstract

Real-time 3D reconstruction of surgical scenes plays a vital role in computer-assisted surgery, holding a promise to enhance surgeons' visibility. Recent advancements in 3D Gaussian Splatting (3DGS) have shown great potential for real-time novel view synthesis of general scenes, which relies on accurate poses and point clouds generated by Structure-from-Motion (SfM) for initialization. However, 3DGS with SfM fails to recover accurate camera poses and geometry in surgical scenes due to the challenges of minimal textures and photometric inconsistencies. To tackle this problem, in this paper, we propose the first SfM-free 3DGS-based method for surgical scene reconstruction by jointly optimizing the camera poses and scene representation. Based on the video continuity, the key of our method is to exploit the immediate optical flow priors to guide the projection flow derived from 3D Gaussians. Unlike most previous methods relying on photometric loss only, we formulate the pose estimation problem as minimizing the flow loss between the projection flow and optical flow. A consistency check is further introduced to filter the flow outliers by detecting the rigid and reliable points that satisfy the epipolar geometry. During 3D Gaussian optimization, we randomly sample frames to optimize the scene representations to grow the 3D Gaussian progressively. Experiments on the SCARED dataset demonstrate our superior performance over existing methods in novel view synthesis and pose estimation with high efficiency. Code is available at https://github.com/wrld/Free-SurGS.

Figures (4)

• Figure 1: 3DGS kerbl20233d meets a major limitation in its reliance on SfM. We propose Free-SurGS to eliminate this need and demonstrate better performance.
• Figure 2: Overview of our proposed Free-SurGS. Given endoscopic monocular images as input, we jointly estimate the camera poses and optimize the 3D Gaussians iteratively by progressive growing.
• Figure 3: Illustration of our proposed flow-induced pose estimation. (a) The consistency check is introduced to filter out the outliers in the optical flow map $\mathbf{O}_{t-1\rightarrow t}$ to obtain reliable and robust correspondences. (b) We formulate the pose estimation problem as matching the projection flow with the optical flow, to compensate for the limitations of photometric loss.
• Figure 4: Qualitative results of novel view synthesis and pose estimation.