EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model

TL;DR

EndoSfM3D tackles robust 3D reconstruction in endoscopic surgery by jointly estimating depth, ego-motion, and intrinsic camera parameters from monocular video within a self-supervised framework. It adapts the Depth Anything V2 foundation model using parameter-efficient fine-tuning (DoRA) and introduces a Pose-Intrinsics Net with attention to predict $K$, $R$, and $t$ alongside depth, guided by a reprojection-based SSL objective and a Tikhonov regularizer. On SCARED and C3VD, the method achieves state-of-the-art depth accuracy (AbsRel around $0.050$–$0.058$) and accurate intrinsics (focal lengths within $2\%$ and principal points within $10\%$), validating its effectiveness under continuous zoom and rotation. This joint, calibration-free approach enhances 3D scene reconstruction, enabling AR visualization and image-guided surgery without physical calibration targets, and paves the way for real-time clinical deployment.

Abstract

3D reconstruction of endoscopic surgery scenes plays a vital role in enhancing scene perception, enabling AR visualization, and supporting context-aware decision-making in image-guided surgery. A critical yet challenging step in this process is the accurate estimation of the endoscope's intrinsic parameters. In real surgical settings, intrinsic calibration is hindered by sterility constraints and the use of specialized endoscopes with continuous zoom and telescope rotation. Most existing methods for endoscopic 3D reconstruction do not estimate intrinsic parameters, limiting their effectiveness for accurate and reliable reconstruction. In this paper, we integrate intrinsic parameter estimation into a self-supervised monocular depth estimation framework by adapting the Depth Anything V2 (DA2) model for joint depth, pose, and intrinsics prediction. We introduce an attention-based pose network and a Weight-Decomposed Low-Rank Adaptation (DoRA) strategy for efficient fine-tuning of DA2. Our method is validated on the SCARED and C3VD public datasets, demonstrating superior performance compared to recent state-of-the-art approaches in self-supervised monocular depth estimation and 3D reconstruction. Code and model weights can be found in project repository: https://github.com/MOYF-beta/EndoSfM3D.

Figures (4)

• Figure 1: Illustration of the proposed EndoSfM3D framework. A ViT-based encoder and DPT-like decoder pre-trained from Depth Anything V2 yang2024depth are used for depth estimation. Weight-Decomposed Low-Rank Adaptation (DoRA) is applied to the feature extractor for parameter-efficient fine-tuning (PEFT). A ResNet-based Pose Encoder with multi-head attention layers extracts SfM features shared between two frames for transformation and intrinsic estimation. Both depth and pose networks are trained using the reprojection loss from MonoDepth2 godard2019digging and the Tikhonov regularizer from AF-SfMLearner shao2022self.
• Figure 2: Results comparison on SCARED for previous researches and our results.
• Figure 3: Comparison of 3D reconstruction results using AF-SfMLearnershao2022self, DARESzeinoddin2024dares, and our method.
• Figure 4: Trajectory estimation results on 2 SCARED sequences.