Sim2Real in endoscopy segmentation with a novel structure aware image translation

TL;DR

This paper tackles the sim-to-real gap in endoscopy image segmentation by pairing synthetic data with a depth-consistency constrained image translation, enabling realistic textures to be added without altering key scene layout. The pipeline consists of (i) automatic labeling of VR-CAPS-colon data via 3D mesh curvature, (ii) a CycleGAN-based translation augmented with depth-consistency to preserve structure, and (iii) a downstream segmentation model based on TransUNet with the EndoFM backbone trained on translated data. The authors demonstrate substantial improvements in fold-segmentation performance when training solely on translated synthetic data and releasing a new automatically labeled fold-segmentation benchmark. The work advances practical sim-to-real transfer for endoscopy and can extend to other landmarks like polyps, potentially accelerating training data generation for medical image analysis.

Abstract

Automatic segmentation of anatomical landmarks in endoscopic images can provide assistance to doctors and surgeons for diagnosis, treatments or medical training. However, obtaining the annotations required to train commonly used supervised learning methods is a tedious and difficult task, in particular for real images. While ground truth annotations are easier to obtain for synthetic data, models trained on such data often do not generalize well to real data. Generative approaches can add realistic texture to it, but face difficulties to maintain the structure of the original scene. The main contribution in this work is a novel image translation model that adds realistic texture to simulated endoscopic images while keeping the key scene layout information. Our approach produces realistic images in different endoscopy scenarios. We demonstrate these images can effectively be used to successfully train a model for a challenging end task without any real labeled data. In particular, we demonstrate our approach for the task of fold segmentation in colonoscopy images. Folds are key anatomical landmarks that can occlude parts of the colon mucosa and possible polyps. Our approach generates realistic images maintaining the shape and location of the original folds, after the image-style-translation, better than existing methods. We run experiments both on a novel simulated dataset for fold segmentation, and real data from the EndoMapper (EM) dataset. All our new generated data and new EM metadata is being released to facilitate further research, as no public benchmark is currently available for the task of fold segmentation.

Figures (8)

• Figure 1: Overview of the proposed new translation approach and subsequent downstream task. During training, our pipeline uses real (x) and synthetic (y) data to learn a model for image translation, to obtain synthetic data with realistic texture (z) while maintaining the most relevant layout structure through depth consistency. The downstream task model can then be trained supervised using the generated data. During inference, the model previously trained on our generated data is applied directly to real images.
• Figure 2: Labeled data generation. (a) Simulated colon from VR-CAPS, (b) curvature principal directions, and (c) binary segmentation. (d) Simulated images from VR-CAPS with corresponding (e) binary masks (f) instance masks and (g) depth maps.
• Figure 3: Simulated colonoscopy images processed with different style transfer models to have realistic texture. CycleGAN, I2I and Ours are trained on 1400 simulated images from VR-CAPS and 1400 real images from EM datasets. $StyTR^2$ and AdaIN models are used as provided, pretrained, and only take one target style input example. SCSfMLearner bian2021unsupervised is used pretrained on VR-CAPS incetan2020vrcaps for depth inference in our approach.
• Figure 4: Examples from three different datasets used as source style in our experiments. (a) Colonoscopy images from EM azagra2023endomapper. (b) NBI colonoscopy images from EM azagra2023endomapper. (c) Laparoscopy images from Cholec80 twinanda2016endonet.
• Figure 5: Style transfer using different approaches. (a) All models trained on 1400 simulated colonoscopy images from VR-CAPS and 3640 real NBI colonoscopy images from EM . (b) CycleGAN, I2I* and Ours trained on 2000 simulated laparoscopic images pfeiffer2019generating and 2840 real laparoscopic images from Cholec80 twinanda2016endonet. I2I corresponds to results published in pfeiffer2019generating trained on 20000 simulated images pfeiffer2019generating and 74 000 real images from Cholec80 twinanda2016endonet. In both experiments, DepthAnything depthanything is used for depth inference in our approach.