Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open-Source Multi-Viewpoint Surgical Telerobotics

Guido Caccianiga, Yarden Sharon, Bernard Javot, Senya Polikovsky, Gökce Ergün, Ivan Capobianco, André L. Mihaljevic, Anton Deguet, Katherine J. Kuchenbecker

TL;DR

Open-Source Multi-Viewpoint Surgical Telerobotics addresses the limitation of single-view MIS systems by proposing a synchronized, multi-endoscope platform built on the dVRK. The approach combines a high-performance, low-latency stereo vision pipeline with hardware and software modifications to enable two independent viewpoints and potential autonomous assistance, including multi-console collaboration. The authors plan to open-source the hardware, software, and datasets to accelerate research, enabling robust 3D intraoperative perception and scalable collaboration. This work aims to improve depth perception, collaborative teleoperation, and clinical translation by providing rich, synchronized multi-view data and extensible robotic control architectures.

Abstract

As robots for minimally invasive surgery (MIS) gradually become more accessible and modular, we believe there is a great opportunity to rethink and expand the visualization and control paradigms that have characterized surgical teleoperation since its inception. We conjecture that introducing one or more additional adjustable viewpoints in the abdominal cavity would not only unlock novel visualization and collaboration strategies for surgeons but also substantially boost the robustness of machine perception toward shared autonomy. Immediate advantages include controlling a second viewpoint and teleoperating surgical tools from a different perspective, which would allow collaborating surgeons to adjust their views independently and still maneuver their robotic instruments intuitively. Furthermore, we believe that capturing synchronized multi-view 3D measurements of the patient's anatomy would unlock advanced scene representations. Accurate real-time intraoperative 3D perception will allow algorithmic assistants to directly control one or more robotic instruments and/or robotic cameras. Toward these goals, we are building a synchronized multi-viewpoint, multi-sensor robotic surgery system by integrating high-performance vision components and upgrading the da Vinci Research Kit control logic. This short paper reports a functional summary of our setup and elaborates on its potential impacts in research and future clinical practice. By fully open-sourcing our system, we will enable the research community to reproduce our setup, improve it, and develop powerful algorithms, effectively boosting clinical translation of cutting-edge research.

Open-Source Multi-Viewpoint Surgical Telerobotics

TL;DR

Open-Source Multi-Viewpoint Surgical Telerobotics addresses the limitation of single-view MIS systems by proposing a synchronized, multi-endoscope platform built on the dVRK. The approach combines a high-performance, low-latency stereo vision pipeline with hardware and software modifications to enable two independent viewpoints and potential autonomous assistance, including multi-console collaboration. The authors plan to open-source the hardware, software, and datasets to accelerate research, enabling robust 3D intraoperative perception and scalable collaboration. This work aims to improve depth perception, collaborative teleoperation, and clinical translation by providing rich, synchronized multi-view data and extensible robotic control architectures.

Abstract

As robots for minimally invasive surgery (MIS) gradually become more accessible and modular, we believe there is a great opportunity to rethink and expand the visualization and control paradigms that have characterized surgical teleoperation since its inception. We conjecture that introducing one or more additional adjustable viewpoints in the abdominal cavity would not only unlock novel visualization and collaboration strategies for surgeons but also substantially boost the robustness of machine perception toward shared autonomy. Immediate advantages include controlling a second viewpoint and teleoperating surgical tools from a different perspective, which would allow collaborating surgeons to adjust their views independently and still maneuver their robotic instruments intuitively. Furthermore, we believe that capturing synchronized multi-view 3D measurements of the patient's anatomy would unlock advanced scene representations. Accurate real-time intraoperative 3D perception will allow algorithmic assistants to directly control one or more robotic instruments and/or robotic cameras. Toward these goals, we are building a synchronized multi-viewpoint, multi-sensor robotic surgery system by integrating high-performance vision components and upgrading the da Vinci Research Kit control logic. This short paper reports a functional summary of our setup and elaborates on its potential impacts in research and future clinical practice. By fully open-sourcing our system, we will enable the research community to reproduce our setup, improve it, and develop powerful algorithms, effectively boosting clinical translation of cutting-edge research.
Paper Structure (6 sections, 2 figures)

This paper contains 6 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Clinical Applications
  3. Proposed Technology
  4. Multi-endoscope Vision Pipeline
  5. Multi-endoscope Surgical Robot
  6. Impact and Future work

Figures (2)

  • Figure 1: Our multi-viewpoint surgical robot (dVRK) featuring two standard instruments (PSM) and two camera manipulators (ECM); each ECM holds one of our synchronized, low-latency, stereo endoscope prototypes.
  • Figure 2: a) Our stereo endoscope prototype. Both images are captured by a single large global-shutter sensor. b) Our processing pipeline. Images travel directly from the camera to the graphics card (GPUDirect RDMA) to be recorded (GPUDirect Storage) and visualized (G-Sync) with minimal latency. Other sensors (e.g., lidar) can be added via the external FPGA-MIPI bridge. A network switch manages clock synchronization (PTP) across the vision pipeline and the surgical robot (dVRK). c) The components of the customized dVRK and its potential expansions to collaborative multi-console setups.