LightNeuS: Neural Surface Reconstruction in Endoscopy using Illumination Decline

TL;DR

LightNeuS addresses dense 3D reconstruction of endoluminal surfaces from monocular endoscopy by leveraging two endoscopy-specific priors: the watertight nature of internal cavities via a signed distance function and the strong inverse-square illumination decline due to the co-located light source. It extends NeuS by providing the light-distance $t$ to the renderer and by incorporating a calibrated endoscope photometric model, making brightness a useful depth cue through the $1/t^2$ term. On the C3VD phantom dataset, it achieves millimeter-scale reconstructions and enables dense, watertight models of long colon sections, including plausible extrapolation into unseen regions. This work enables automatic quality assessment of colon exploration by quantifying observed mucosa coverage and lays groundwork for future real-time or near-real-time endoscopic reconstruction capabilities.

Abstract

We propose a new approach to 3D reconstruction from sequences of images acquired by monocular endoscopes. It is based on two key insights. First, endoluminal cavities are watertight, a property naturally enforced by modeling them in terms of a signed distance function. Second, the scene illumination is variable. It comes from the endoscope's light sources and decays with the inverse of the squared distance to the surface. To exploit these insights, we build on NeuS, a neural implicit surface reconstruction technique with an outstanding capability to learn appearance and a SDF surface model from multiple views, but currently limited to scenes with static illumination. To remove this limitation and exploit the relation between pixel brightness and depth, we modify the NeuS architecture to explicitly account for it and introduce a calibrated photometric model of the endoscope's camera and light source. Our method is the first one to produce watertight reconstructions of whole colon sections. We demonstrate excellent accuracy on phantom imagery. Remarkably, the watertight prior combined with illumination decline, allows to complete the reconstruction of unseen portions of the surface with acceptable accuracy, paving the way to automatic quality assessment of cancer screening explorations, measuring the global percentage of observed mucosa.

Figures (7)

• Figure 1: From NeuS to LightNeuS. The original NeuS architecture is depicted by the black arrows. In LightNeuS, when training the network with a sampled point, we provide the sampling distance $t$ to the renderer, that takes into account illumination decline. We also incorporate a calibrated photometric endoscope model that is used to correctly compute the photometric loss. The changes are shown in red.
• Figure 2: Benefits of illumination decline. Result on the "Cecum 1 a" sequence. Top: The NeuS reconstruction exhibits multiple artifacts that make it unusable. Bottom: Our reconstruction is much closer to the ground truth shape. The error is shown in blue if the reconstruction is inside the surface, and in red otherwise. A fully saturated red or blue denotes an error of more than 1cm and grey denotes no error at all.
• Figure 3: Reconstructing partially observed regions. Results on "Transcending 4 a" sequence. The camera performs a short trajectory from (a) to (b). In (c) we represent both frames and intermediate camera poses. (d) Number of frames seeing each surface point, with GT unobserved areas shown in gray. (e) We managed to reconstruct a curved section of the colon. (f) Our method plausibly estimates the wall of the colon at the right of camera (b), although it was never seen in the images.
• Figure 4: Reconstructing with low parallax. Results on "Transcending 1 a" sequence. (c) The camera travels in a straight line, covering less than a third of the section. As shown in (a) and (b), the haustra completely hide the background walls. (e) Consequently, the reconstruction underestimates the diameter of the end of the tube. However, the three characteristic folds in our reconstructed colon match the ground truth in number and location. In addition, areas observed multiple times ---red in (d)--- are reconstructed with high accuracy ---gray in (f).
• Figure 5: Reconstruction convergence. Results on "Transcending 1 a" sequence. We show the intermediate results for N optimisation iterations. We see how the reconstruction converges quickly. In 65k iterations we already have a reasonable solution, compared to the 300k iterations proposed by the authors of NeuS.