Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments

TL;DR

This paper introduces 2D and 3D learning-based perception algorithms and develops a physically-realistic simulator that models flexible instruments dynamics and shows that these algorithms improve control of flexible instruments, reducing manipulation time for trajectory-following tasks and enhancing understanding of surgical scenarios, leading to robust endoluminal surgeries.

Abstract

Endoluminal surgery offers a minimally invasive option for early-stage gastrointestinal and urinary tract cancers but is limited by surgical tools and a steep learning curve. Robotic systems, particularly continuum robots, provide flexible instruments that enable precise tissue resection, potentially improving outcomes. This paper presents a visual perception platform for a continuum robotic system in endoluminal surgery. Our goal is to utilize monocular endoscopic image-based perception algorithms to identify position and orientation of flexible instruments and measure their distances from tissues. We introduce 2D and 3D learning-based perception algorithms and develop a physically-realistic simulator that models flexible instruments dynamics. This simulator generates realistic endoluminal scenes, enabling control of flexible robots and substantial data collection. Using a continuum robot prototype, we conducted module and system-level evaluations. Results show that our algorithms improve control of flexible instruments, reducing manipulation time by over 70% for trajectory-following tasks and enhancing understanding of surgical scenarios, leading to robust endoluminal surgeries.

Figures (16)

• Figure 1: Robotic instrument configurations and endoscopic scenarios in continuum robotic endoluminal surgery. (a) For transurethral bladder tumour resection, the robotic instruments are deployed through a standard urology outer sheath alongside a standard telescope. (b) Two external instrument channels equipped to a standard single-channel GI endoscope (e.g. from Olympus) to deliver the robotic instruments. (c) Monocular endoluminal image captured by GI endoscope.
• Figure 2: Overview of the proposed image-based perception framework for continuum robotic endoluminal surgery. The framework consists of three modules in its hybrid training: 1) 2D continuum robot segmentation module for flexible instruments recognition; 2) 3D robot state estimation module for flexible instrument shape calculation ; and 3) 3D depth estimation module for geometric information provision about the entire endoscopic scene. Monocular endoscopic images serve as the sole input for the perception framework. After training, the framework is deployed on a novel continuum robotic system.
• Figure 3: Calculation of the Per-Pixel Rendering ($\textbf{PPR}$) field from the depth estimation, which is the process of PPR calculation illustrated in Fig. \ref{['fig:method']}. The $\textbf{PPR}$ field shows a strong correlation with the corresponding input endoscopic image. Arrows from Depth indicate how depth is transformed into attenuation and light direction. Additionally, arrows between Pseudo and Real images represent the correlation computation, with the correlation values normalized to a range of 0 to 1.
• Figure 4: Monocular depth estimation model for continuum robotic endoluminal surgical scene.
• Figure 5: Physically-realistic simulation of continuum robotic system for endoluminal surgery. (a) Setup of the continuum robotic system. (b) Configuration of flexible instruments approximated by curve segments in camera coordinate system. The flexible robot consists of three segments: Shoulder $\widehat{\textbf{OS}}$, Proximal $\widehat{\textbf{SP}}$, and Distal $\widehat{\textbf{PD}}$. $\alpha$ and $\beta$ denote internal bending angles of $\widehat{\textbf{SP}}$ and $\widehat{\textbf{PD}}$. $\gamma$ and $\delta$ represent $\widehat{\textbf{OS}}$ yaw and roll angles. (c) Comparison of real and synthetic endoscopic images. More comparison results are shown in Video 1.