Structure-preserving Image Translation for Depth Estimation in Colonoscopy Video

TL;DR

The paper tackles the domain gap in colonoscopy depth estimation by introducing a structure-preserving sim2real image translation that converts synthetic, depth-annotated frames into realistic-looking images while preserving depth geometry. It employs a CycleGAN framework augmented with a mutual-information depth-preservation loss, enabling supervised depth learning from translated data and evaluation on real clinical-like frames. Two hand-picked clinical-viewpoint datasets (oblique and en-face) are introduced to guide translation and assess depth-estimation performance, with experiments on C3VD showing improved zero-shot depth accuracy and more faithful geometric reconstructions. The work provides code and data resources and holds practical potential to enhance 3D reconstruction, navigation, and automatic measurements in colonoscopy.

Abstract

Monocular depth estimation in colonoscopy video aims to overcome the unusual lighting properties of the colonoscopic environment. One of the major challenges in this area is the domain gap between annotated but unrealistic synthetic data and unannotated but realistic clinical data. Previous attempts to bridge this domain gap directly target the depth estimation task itself. We propose a general pipeline of structure-preserving synthetic-to-real (sim2real) image translation (producing a modified version of the input image) to retain depth geometry through the translation process. This allows us to generate large quantities of realistic-looking synthetic images for supervised depth estimation with improved generalization to the clinical domain. We also propose a dataset of hand-picked sequences from clinical colonoscopies to improve the image translation process. We demonstrate the simultaneous realism of the translated images and preservation of depth maps via the performance of downstream depth estimation on various datasets.

Figures (6)

• Figure 1: Sample frames from various datasets.
• Figure 2: Image translation framework with image domains $A$ and $B$, generators $G:A \rightarrow B$ and $F: B \rightarrow A$, and discriminators $D_A$ and $D_B$. Let $a \in A, b \in B$ denote data samples and let $\hat{D}_a$ denote the depth map corresponding to sample $a$. Downstream depth estimation uses output of generator $G(A)$.
• Figure 3: Examples comparing the SimCol3D input frame, our translation, closest image in oblique dataset via SSIM, and translation with vanilla CycleGAN.
• Figure 4: Depth estimation on oblique dataset. Boxes highlight differences. Image translation framework improves monocular depth estimation in general, with best performance using our proposed dataset as translation target.
• Figure 6: Depth estimation on additional oblique examples.