Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online 3D reconstruction and dense tracking in endoscopic videos

Michel Hayoz, Christopher Hahne, Thomas Kurmann, Max Allan, Guido Beldi, Daniel Candinas, ablo Márquez-Neila, Raphael Sznitman

TL;DR

This work introduces an efficient online fitting algorithm that optimizes the scene parameters, enabling consistent tracking and accurate reconstruction, and enables various downstream applications thus contributing to advancing the capabilities of surgical assistance systems.

Abstract

3D scene reconstruction from stereo endoscopic video data is crucial for advancing surgical interventions. In this work, we present an online framework for online, dense 3D scene reconstruction and tracking, aimed at enhancing surgical scene understanding and assisting interventions. Our method dynamically extends a canonical scene representation using Gaussian splatting, while modeling tissue deformations through a sparse set of control points. We introduce an efficient online fitting algorithm that optimizes the scene parameters, enabling consistent tracking and accurate reconstruction. Through experiments on the StereoMIS dataset, we demonstrate the effectiveness of our approach, outperforming state-of-the-art tracking methods and achieving comparable performance to offline reconstruction techniques. Our work enables various downstream applications thus contributing to advancing the capabilities of surgical assistance systems.

Online 3D reconstruction and dense tracking in endoscopic videos

TL;DR

This work introduces an efficient online fitting algorithm that optimizes the scene parameters, enabling consistent tracking and accurate reconstruction, and enables various downstream applications thus contributing to advancing the capabilities of surgical assistance systems.

Abstract

3D scene reconstruction from stereo endoscopic video data is crucial for advancing surgical interventions. In this work, we present an online framework for online, dense 3D scene reconstruction and tracking, aimed at enhancing surgical scene understanding and assisting interventions. Our method dynamically extends a canonical scene representation using Gaussian splatting, while modeling tissue deformations through a sparse set of control points. We introduce an efficient online fitting algorithm that optimizes the scene parameters, enabling consistent tracking and accurate reconstruction. Through experiments on the StereoMIS dataset, we demonstrate the effectiveness of our approach, outperforming state-of-the-art tracking methods and achieving comparable performance to offline reconstruction techniques. Our work enables various downstream applications thus contributing to advancing the capabilities of surgical assistance systems.
Paper Structure (10 sections, 8 equations, 3 figures, 3 tables)

This paper contains 10 sections, 8 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Scene Model
  4. Canonical and non-rigid scenes.
  5. Rendering.
  6. Model fitting
  7. Experimental setup and results
  8. Results
  9. Conclusion
  10. Disclosure of Interests.

Figures (3)

  • Figure 1: Overview of our proposed scene reconstruction and dense tracking method.
  • Figure 2: 2D point tracking over time results. Annotated ground-truth points are marked with triangles, PIPS++ with squares, and ours with crosses.
  • Figure 3: 3D semantic segmentation as a downstream application. Semantic classes are overlayed: gall-bladder (purple), liver (red) and plastic tubes (yellow).