Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy

Laura Salort-Benejam, Antonio Agudo

TL;DR

Inspired by neural rendering, NeRFscopy is introduced, a self-supervised pipeline for novel view synthesis and 3D reconstruction of deformable endoscopic tissues from a monocular video that achieves accurate results in terms of novel view synthesis, outperforming competing methods across various challenging endoscopy scenes.

Abstract

Endoscopy is essential in medical imaging, used for diagnosis, prognosis and treatment. Developing a robust dynamic 3D reconstruction pipeline for endoscopic videos could enhance visualization, improve diagnostic accuracy, aid in treatment planning, and guide surgery procedures. However, challenges arise due to the deformable nature of the tissues, the use of monocular cameras, illumination changes, occlusions and unknown camera trajectories. Inspired by neural rendering, we introduce NeRFscopy, a self-supervised pipeline for novel view synthesis and 3D reconstruction of deformable endoscopic tissues from a monocular video. NeRFscopy includes a deformable model with a canonical radiance field and a time-dependent deformation field parameterized by SE(3) transformations. In addition, the color images are efficiently exploited by introducing sophisticated terms to learn a 3D implicit model without assuming any template or pre-trained model, solely from data. NeRFscopy achieves accurate results in terms of novel view synthesis, outperforming competing methods across various challenging endoscopy scenes.

NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy

TL;DR

Inspired by neural rendering, NeRFscopy is introduced, a self-supervised pipeline for novel view synthesis and 3D reconstruction of deformable endoscopic tissues from a monocular video that achieves accurate results in terms of novel view synthesis, outperforming competing methods across various challenging endoscopy scenes.

Abstract

Endoscopy is essential in medical imaging, used for diagnosis, prognosis and treatment. Developing a robust dynamic 3D reconstruction pipeline for endoscopic videos could enhance visualization, improve diagnostic accuracy, aid in treatment planning, and guide surgery procedures. However, challenges arise due to the deformable nature of the tissues, the use of monocular cameras, illumination changes, occlusions and unknown camera trajectories. Inspired by neural rendering, we introduce NeRFscopy, a self-supervised pipeline for novel view synthesis and 3D reconstruction of deformable endoscopic tissues from a monocular video. NeRFscopy includes a deformable model with a canonical radiance field and a time-dependent deformation field parameterized by SE(3) transformations. In addition, the color images are efficiently exploited by introducing sophisticated terms to learn a 3D implicit model without assuming any template or pre-trained model, solely from data. NeRFscopy achieves accurate results in terms of novel view synthesis, outperforming competing methods across various challenging endoscopy scenes.
Paper Structure (4 sections, 5 equations, 4 figures, 3 tables)

This paper contains 4 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. NeRF for dynamic endoscopy tissues
  3. Experimental Evaluation
  4. Conclusion

Figures (4)

  • Figure 1: Overview of our NeRFscopy pipeline for time-varying NeRF from RGB endoscopy videos.
  • Figure 2: Qualitative evaluation of the depth regularizer on the lung lobectomy video. Columns show from left to right: arbitrary input frame, RGB rendered image, input depth estimation and depth rendered result. For depth initialization, we consider DPT DPT (top), IID-SfmLearner IID_SfmLearner (middle) or Depth-Anything depthanything (bottom).
  • Figure 3: Qualitative evaluation on real videos. Columns show from left to right: arbitrary input frame, RGB rendering, input depth estimation, depth rendering, and side 3D view. From top to bottom: TECAB1, TECAB2, lung lobectomy and bronchoscopy images.
  • Figure 4: Rendered results for five consecutive frames. Novel views highlighted in red. From top to bottom: TECAB1, TECAB2, lung lobectomy and bronchoscopy images.