Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images

TL;DR

This work tackles depth estimation in colonoscopy by learning Shape-from-Shading from a realistic synthetic colonoscopy database. It introduces SfSNet, an encoder–decoder CNN that regresses pixel-wise depth from a single RGB frame under Lambertian assumptions, guided by a custom loss that emphasizes edges and curvature. A fully synthetic dataset with pixel-wise depth annotations is released and used to train two learning strategies—Traditional and Curriculum Learning—demonstrating that curriculum-based training improves fine-grained depth details and convergence, with EfficientNetB0 achieving RMSE as low as $0.451\,\text{cm}$ and ThAcc up to $96.13\%$. Qualitative evaluation on real colonoscopy images confirms coherent 3D reconstructions, and the synthetic data enables robust benchmarking and potential extensions to temporal consistency and camera-pose estimation.

Abstract

Colonoscopy is the choice procedure to diagnose colon and rectum cancer, from early detection of small precancerous lesions (polyps), to confirmation of malign masses. However, the high variability of the organ appearance and the complex shape of both the colon wall and structures of interest make this exploration difficult. Learned visuospatial and perceptual abilities mitigate technical limitations in clinical practice by proper estimation of the intestinal depth. This work introduces a novel methodology to estimate colon depth maps in single frames from monocular colonoscopy videos. The generated depth map is inferred from the shading variation of the colon wall with respect to the light source, as learned from a realistic synthetic database. Briefly, a classic convolutional neural network architecture is trained from scratch to estimate the depth map, improving sharp depth estimations in haustral folds and polyps by a custom loss function that minimizes the estimation error in edges and curvatures. The network was trained by a custom synthetic colonoscopy database herein constructed and released, composed of 248,400 frames (47 videos), with depth annotations at the level of pixels. This collection comprehends 5 subsets of videos with progressively higher levels of visual complexity. Evaluation of the depth estimation with the synthetic database reached a threshold accuracy of 95.65%, and a mean-RMSE of 0.451 cm, while a qualitative assessment with a real database showed consistent depth estimations, visually evaluated by the expert gastroenterologist coauthoring this paper. Finally, the method achieved competitive performance with respect to another state-of-the-art method using a public synthetic database and comparable results in a set of images with other five state-of-the-art methods.

Figures (13)

• Figure 1: Pipeline of the proposed approach. First, a synthetic colonoscopy database is constructed (see Section \ref{['subsec:synthetic_colon_db']}). Then, a Shape-from-Shading strategy estimates depth map (see Section \ref{['subsec:depth_estimation']}). Finally, a depth estimation model trained in synthetic data is applied in real colonoscopy images.
• Figure 2: Overview of the convolutional network, built as an encoder-decoder architecture with skip connections. An RGB input image is used to apply a pixel-wise regression to obtain a scalar depth map with half the spatial resolution.
• Figure 5: A structure matching anatomical colon measures and a set of spheres with sizes from $30 \: mm$ to $5 \: mm$ are the basic colon and polyp models, respectively.
• Figure 6: Frames with depth annotations for each level of the synthetic video collection. Levels 1 to 4 are shown in the top panel, where synthetic frames are in the first row along with their corresponding depth maps in second row. Bottom panel shows four pictures corresponding to level 5. A gray-scale bar at the right of depth map rows shows the corresponding depth values in centimeters.
• Figure 7: Depth map of a synthetic image is compared with estimations from the two learning strategies on EfficientNetB0. A zoomed view of the region of interest shows the estimation of a medium-size polyp and caecum-appendix aperture.