Sim2Real within 5 Minutes: Efficient Domain Transfer with Stylized Gaussian Splatting for Endoscopic Images

TL;DR

This work tackles the domain gap between pre-operative CT-derived structures and intra-operative endoscopic images by introducing stylized Gaussian splatting. It uses differentiable Gaussian point clouds as a structural prior and confines style transfer to color-appearance parameters, guided by global, local, and structure-consistency losses to preserve content. The approach achieves near real-time training (about five minutes) with strong image translation metrics and improved downstream pose estimation, outperforming several baselines even with very limited real data. This enables practical, sensor-free, vision-based navigation for endoluminal interventions by providing fast, reliable sim-to-real transfers.

Abstract

Robot assisted endoluminal intervention is an emerging technique for both benign and malignant luminal lesions. With vision-based navigation, when combined with pre-operative imaging data as priors, it is possible to recover position and pose of the endoscope without the need of additional sensors. In practice, however, aligning pre-operative and intra-operative domains is complicated by significant texture differences. Although methods such as style transfer can be used to address this issue, they require large datasets from both source and target domains with prolonged training times. This paper proposes an efficient domain transfer method based on stylized Gaussian splatting, only requiring a few of real images (10 images) with very fast training time. Specifically, the transfer process includes two phases. In the first phase, the 3D models reconstructed from CT scans are represented as differential Gaussian point clouds. In the second phase, only color appearance related parameters are optimized to transfer the style and preserve the visual content. A novel structure consistency loss is applied to latent features and depth levels to enhance the stability of the transferred images. Detailed validation was performed to demonstrate the performance advantages of the proposed method compared to that of the current state-of-the-art, highlighting the potential for intra-operative surgical navigation.

Figures (5)

• Figure 1: Challenges in efficient style transfer. Given sufficient training data and time, existing style transfer networks are capable of producing high-quality images. However, in the absence of adequate training data, the style transfer method may generate target-like textures, but introduces noticeable content distortion. Additionally, with insufficient training time, the style transfer method tends to produce blurry images. Our method is capable of achieving high-quality style transfer even with limited samples and very fast training time.
• Figure 2: The overall framework of our proposed method. In the differential representation phase, based on virtual images and poses, the Gaussian point clouds can be built based on Gaussian splatting theory. In the efficient transfer phase, the parameters of Gaussian point clouds are decomposed by structure-related parameters and color appearance-related parameters. Only color appearance-related parameters are optimized during this phase.
• Figure 3: Examples of the virtual inputs, target real images, and the generated results using different methods. It is evident that GAN-based methods exhibited content bias, StyleRF generated blurry images, while our method effectively transferred the overall style and preserved the input structural content.
• Figure 4: Qualitative results on three sequences. Due to the lack of training images, the images generated by the baselines exhibit structural deviations, resulting in suboptimal performance in pose estimation. Our method preserves the structural content, achieving superior results in pose estimation.
• Figure 5: Visual analysis of the effects of the global loss and the local loss.