General-purpose foundation models for increased autonomy in robot-assisted surgery

TL;DR

It is argued that surgical robots are uniquely positioned to benefit from general-purpose models and provide four guiding actions towards increased autonomy in robot-assisted surgery.

Abstract

The dominant paradigm for end-to-end robot learning focuses on optimizing task-specific objectives that solve a single robotic problem such as picking up an object or reaching a target position. However, recent work on high-capacity models in robotics has shown promise toward being trained on large collections of diverse and task-agnostic datasets of video demonstrations. These models have shown impressive levels of generalization to unseen circumstances, especially as the amount of data and the model complexity scale. Surgical robot systems that learn from data have struggled to advance as quickly as other fields of robot learning for a few reasons: (1) there is a lack of existing large-scale open-source data to train models, (2) it is challenging to model the soft-body deformations that these robots work with during surgery because simulation cannot match the physical and visual complexity of biological tissue, and (3) surgical robots risk harming patients when tested in clinical trials and require more extensive safety measures. This perspective article aims to provide a path toward increasing robot autonomy in robot-assisted surgery through the development of a multi-modal, multi-task, vision-language-action model for surgical robots. Ultimately, we argue that surgical robots are uniquely positioned to benefit from general-purpose models and provide three guiding actions toward increased autonomy in robot-assisted surgery.

Figures (3)

• Figure 1: An architecture diagram of the proposed vision-language-action robot transformer. Video frames are taken as input, flattened, and passed through a linear projection to be used as input tokens along with a word embedding. The transformer encoder outputs action tokens which are de-tokenized to produce a robot action, from which the robot end-effector position is updated.
• Figure 2: A proposed control loop for the autonomous robot transformer-RAS (RT-RAS). Surgeon provides action commands as text input. The RT-RAS executes these commands while maintaining high confidence, otherwise autonomy is switched to the surgeon.
• Figure 3: Outline of the two step pre-training for the RT-RAS. The first involves fine-tuning a vision-language model on captioned surgical demonstrations (e.g. from surgical training videos). The second involves pre-training a vision-language-action model on surgical demonstrations with kinematics.