Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SuFIA: Language-Guided Augmented Dexterity for Robotic Surgical Assistants

Masoud Moghani, Lars Doorenbos, William Chung-Ho Panitch, Sean Huver, Mahdi Azizian, Ken Goldberg, Animesh Garg

TL;DR

SuFIA introduces a generalist, language-guided augmented dexterity framework for robotic surgery that uses large language models to generate high-level plans and low-level control code, interfacing with a perception module through a lightweight API. By avoiding task-specific training and motion-primitives, SuFIA achieves learning-free autonomous sub-task execution while ensuring safety via re-planning and human-in-the-loop control. The approach is validated in high-fidelity simulation and on a physical dVRK platform across four sub-tasks, with performance demonstrating robustness to domain variation though limited by perception reliability and response-time constraints when using external LLMs. The work suggests that language-guided autonomy can reduce hardware and data barriers, enhance surgeon-robot collaboration, and extend autonomous capabilities in surgical settings.

Abstract

In this work, we present SuFIA, the first framework for natural language-guided augmented dexterity for robotic surgical assistants. SuFIA incorporates the strong reasoning capabilities of large language models (LLMs) with perception modules to implement high-level planning and low-level control of a robot for surgical sub-task execution. This enables a learning-free approach to surgical augmented dexterity without any in-context examples or motion primitives. SuFIA uses a human-in-the-loop paradigm by restoring control to the surgeon in the case of insufficient information, mitigating unexpected errors for mission-critical tasks. We evaluate SuFIA on four surgical sub-tasks in a simulation environment and two sub-tasks on a physical surgical robotic platform in the lab, demonstrating its ability to perform common surgical sub-tasks through supervised autonomous operation under challenging physical and workspace conditions. Project website: orbit-surgical.github.io/sufia

SuFIA: Language-Guided Augmented Dexterity for Robotic Surgical Assistants

TL;DR

SuFIA introduces a generalist, language-guided augmented dexterity framework for robotic surgery that uses large language models to generate high-level plans and low-level control code, interfacing with a perception module through a lightweight API. By avoiding task-specific training and motion-primitives, SuFIA achieves learning-free autonomous sub-task execution while ensuring safety via re-planning and human-in-the-loop control. The approach is validated in high-fidelity simulation and on a physical dVRK platform across four sub-tasks, with performance demonstrating robustness to domain variation though limited by perception reliability and response-time constraints when using external LLMs. The work suggests that language-guided autonomy can reduce hardware and data barriers, enhance surgeon-robot collaboration, and extend autonomous capabilities in surgical settings.

Abstract

In this work, we present SuFIA, the first framework for natural language-guided augmented dexterity for robotic surgical assistants. SuFIA incorporates the strong reasoning capabilities of large language models (LLMs) with perception modules to implement high-level planning and low-level control of a robot for surgical sub-task execution. This enables a learning-free approach to surgical augmented dexterity without any in-context examples or motion primitives. SuFIA uses a human-in-the-loop paradigm by restoring control to the surgeon in the case of insufficient information, mitigating unexpected errors for mission-critical tasks. We evaluate SuFIA on four surgical sub-tasks in a simulation environment and two sub-tasks on a physical surgical robotic platform in the lab, demonstrating its ability to perform common surgical sub-tasks through supervised autonomous operation under challenging physical and workspace conditions. Project website: orbit-surgical.github.io/sufia
Paper Structure (24 sections, 7 figures, 2 tables)

This paper contains 24 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models for Robotics
  4. Surgical Augmented Dexterity
  5. Problem Formulation
  6. SuFIA
  7. LLM Generation and Planning
  8. API Library
  9. Perception
  10. Safety
  11. Experimental Results
  12. Experimental Setup
  13. Tasks and Evaluation Metrics
  14. SuFIA Evaluation
  15. Task Prompt Analysis
  16. ...and 9 more sections

Figures (7)

  • Figure 1: An overview of SuFIA automating the lifting of a suture needle from a surgical site.SuFIA receives commands from a surgeon in natural language and converts them to high-level planning and low-level control code. If a task requires object interaction, SuFIA queries a perception module for object state information and generates low-level trajectories and robot actions accordingly. SuFIA can assist a surgeon with open-ended tasks, such as moving the robot in a desired motion to help complete a surgical task. In times of inefficient information, SuFIA delegates full control back to the surgeon.
  • Figure 2: SuFIA architecture and workflows.SuFIA enables a surgeon to naturally interact with the robot by either asking for a complete sub-task (e.g. "pick up the needle and hand it over to the other arm") or generating a trajectory to help with performing a task (e.g. "move the needle 1 cm to the left"). SuFIA uses limited environmental knowledge in natural language (i.e. System Prompt) and scene understanding from a perception module to directly generate high-level plans and low-level sequences of gripper poses to interact with small-scale surgical objects. If SuFIA encounters difficulty in querying for an object or executing a necessary step to solve the surgical sub-task, it hands the control back to the surgeon for teleoperation.
  • Figure 3: Surgical sub-tasks. (a) Needle Lift: lift a suture needle to a desired height, (b) Needle handover: pick and handover a suture needle, (c) Vessel Dilation: grip the vessel rim and dilate by pulling, (d) Shunt Insertion: insert a soft tube into larger vessel phantom. Best viewed in color.
  • Figure 4: Physical Needle Handover task. (1) Starting workspace configuration. The needle is placed in a fixed position within the workspace, and the gripper positions are randomized. In this stage, the SuFIA LLM planner queries for and identifies the pose of the suture needle, determines which robot arm is closest to it, and plans a trajectory for that robot arm to reach the suture needle. (2) The closest robot arm approaches and grasps the suture needle. (3) The suture needle is lifted to a neutral handover position. At this stage, the SuFIA LLM planner detects the pose of the suture needle at the handover position and plans a trajectory for the second robot arm to approach the needle. (4) The second robot arm descends and grasps the needle, then the first robot arm releases the needle after the second robot arm has grasped it. We provide task videos at https://orbit-surgical.github.io/sufia
  • Figure 5: Needle variations in simulation. We consider five instances of simulated suture needles (N1 - N5) with various sizes and shapes to conduct the generalizability experiment in Orbit-Surgical.
  • ...and 2 more figures