Voice-Interactive Surgical Agent for Multimodal Patient Data Control

TL;DR

<3-5 sentence high-level summary>

Abstract

In robotic surgery, surgeons fully engage their hands and visual attention in procedures, making it difficult to access and manipulate multimodal patient data without interrupting the workflow. To overcome this problem, we propose a Voice-Interactive Surgical Agent (VISA) built on a hierarchical multi-agent framework consisting of an orchestration agent and three task-specific agents driven by Large Language Models (LLMs). These LLM-based agents autonomously plan, refine, validate, and reason to interpret voice commands and execute tasks such as retrieving clinical information, manipulating CT scans, or navigating 3D anatomical models within surgical video. We construct a dataset of 240 user commands organized into hierarchical categories and introduce the Multi-level Orchestration Evaluation Metric (MOEM) that evaluates the performance and robustness at both the command and category levels. Experimental results demonstrate that VISA achieves high stage-level accuracy and workflow-level success rates, while also enhancing its robustness by correcting transcription errors, resolving linguistic ambiguity, and interpreting diverse free-form expressions. These findings highlight the strong potential of VISA to support robotic surgery and its scalability for integrating new functions and agents.

Figures (13)

• Figure 1: Overall framework of the proposed VISA: (a) Workflow orchestrator agent that autonomously plans the order of function execution; (b) Workflow functions that capture, transcribe, correct, and route voice commands; (c) Task-specific agent functions that control patient data on the target video clip; (d) Memory state with local memory for each clip and a global memory for shared contextual understanding across clips.
• Figure 2: Overall workflow planned by the workflow orchestrator agent with a hybrid approach of LLM and mathematical rules.
• Figure 3: The overview of the IR agent, which executes either the show or remove function with inferred state parameters.
• Figure 4: The overview of the IV agent, which executes show, zoom in, zoom out, or remove function with inferred state parameters.
• Figure 5: The overview of the AR agent. The action determination function predicts the action and state parameters, then the AR agent executes static, zoom in, rotate, zoom out, or remove function with predicted parameters.