RealSynCol: a high-fidelity synthetic colon dataset for 3D reconstruction applications

TL;DR

RealSynCol tackles the lack of ground-truth data for endoscopic 3D reconstruction by delivering a high-fidelity synthetic colon dataset generated from CT-derived anatomies with dense ground-truth annotations. The authors present a end-to-end pipeline in Blender and 3D Slicer that yields 20 sequences (28,130 frames) with depth, optical flow, camera trajectories, and meshes, plus clinically inspired motion and texture realism. Through DAM v2 fine-tuning with LoRA and a cross-dataset benchmark against SimCol3D and C3VD, RealSynCol demonstrates improved metric-depth accuracy, reduced reliance on test-time scale, and enhanced generalization to real endoscopic imagery; ablations quantify the impact of texture and reflections. The work provides both a valuable benchmark and practical resources (data and code) to advance endoscopic 3D reconstruction, with implications for improved detection of blind spots and lesion localization in clinical practice.

Abstract

Deep learning has the potential to improve colonoscopy by enabling 3D reconstruction of the colon, providing a comprehensive view of mucosal surfaces and lesions, and facilitating the identification of unexplored areas. However, the development of robust methods is limited by the scarcity of large-scale ground truth data. We propose RealSynCol, a highly realistic synthetic dataset designed to replicate the endoscopic environment. Colon geometries extracted from 10 CT scans were imported into a virtual environment that closely mimics intraoperative conditions and rendered with realistic vascular textures. The resulting dataset comprises 28\,130 frames, paired with ground truth depth maps, optical flow, 3D meshes, and camera trajectories. A benchmark study was conducted to evaluate the available synthetic colon datasets for the tasks of depth and pose estimation. Results demonstrate that the high realism and variability of RealSynCol significantly enhance generalization performance on clinical images, proving it to be a powerful tool for developing deep learning algorithms to support endoscopic diagnosis.

Figures (11)

• Figure 1: Schema of the dataset generation process. CT scans in DICOM files are imported into 3D Slicer FEDOROV2012, segmented, and any noise left is corrected. The centerline is identified and refined in each. The corrected 3D model and the centerline are imported into Blender, where texture, light, and a virtual camera are added. This environment is then used to extract RGB frames, ground truth depth, optical flow, camera parameters, 3D mesh, and trajectory.
• Figure 2: Example of a 3D colon model and the relative centerline in Blender
• Figure 3: Examples of generated images with the relative ground truths. On the left, RGB frames, in the center the depth maps, on the right the optical flow in Middlebury color coding baker2011.
• Figure 4: Examples of depth maps predicted using zero-shot DAM v2 and fine-tuned DAM v2
• Figure 5: Examples of depth maps from the clinical SUN Database SUN predicted using the models included in the ablation study. For clinical images ground truth depth maps are not available.