Surgical Action Planning with Large Language Models

TL;DR

The paper tackles the lack of intraoperative predictive planning in robot-assisted minimally invasive surgery by defining Surgical Action Planning (SAP) and proposing LLM-SAP, a framework that combines Near-History Focus Memory (NHFM) with a prompts factory to predict future actions from visual inputs. SAP outputs a long-horizon action sequence $\,A = {a_1, ..., a_t}$ given a visual history $H$ and a goal $G$ within horizon $T$, using two variants: a text-based IndirNHFM and a vision-language DirNHFM guided by action-planning prompts. The authors introduce the CholecT50-SAP dataset, develop a zero-shot and LoRA-based supervised fine-tuning regime (SFT) with data distilled from GPT-4o, and demonstrate that SFT substantially improves performance across metrics such as SLAcc, VLAcc, and ReAcc, with IndirNHFM often outperforming DirNHFM in zero-shot settings. Ablation studies on the NHFM component show that incorporating prior action labels into the memory yields the strongest results, and future work envisions reasoning-based LLMs and broader procedural and robotic integration for intraoperative planning and education.

Abstract

In robot-assisted minimally invasive surgery, we introduce the Surgical Action Planning (SAP) task, which generates future action plans from visual inputs to address the absence of intraoperative predictive planning in current intelligent applications. SAP shows great potential for enhancing intraoperative guidance and automating procedures. However, it faces challenges such as understanding instrument-action relationships and tracking surgical progress. Large Language Models (LLMs) show promise in understanding surgical video content but remain underexplored for predictive decision-making in SAP, as they focus mainly on retrospective analysis. Challenges like data privacy, computational demands, and modality-specific constraints further highlight significant research gaps. To tackle these challenges, we introduce LLM-SAP, a Large Language Models-based Surgical Action Planning framework that predicts future actions and generates text responses by interpreting natural language prompts of surgical goals. The text responses potentially support surgical education, intraoperative decision-making, procedure documentation, and skill analysis. LLM-SAP integrates two novel modules: the Near-History Focus Memory Module (NHF-MM) for modeling historical states and the prompts factory for action planning. We evaluate LLM-SAP on our constructed CholecT50-SAP dataset using models like Qwen2.5 and Qwen2-VL, demonstrating its effectiveness in next-action prediction. Pre-trained LLMs are tested in a zero-shot setting, and supervised fine-tuning (SFT) with LoRA is implemented. Our experiments show that Qwen2.5-72B-SFT surpasses Qwen2.5-72B with a 19.3% higher accuracy.

Figures (4)

• Figure 1: The architecture of our LLM-SAP. It is developed to predict the next surgical action and form long-horizon action chains by breaking procedures into a fixed set of actions, powered by advanced LLMs. LLM-SAP has two versions: a text-based LLM planning model that utilizes the text descriptions of the visual history in IndirNHFM, and a VLM planning model that uses visual history directly in DirNHFM, both guided by the action planning prompt. The data flow of these two versions is indicated by the pink and yellow lines, respectively. Given the surgical domain knowledge base and action planning prompts like "What's the next action?", LLMs analyze the history memory state to understand the surgical progress and generate structured responses, including progress assessment, safety considerations, and future action recommendations.
• Figure 2: Example of the CholecT50-SAP dataset we constructed.
• Figure 3: (a) Action planning results visualization. White text displays the planning results, showing the Top 3 future action predictions in order. ${ \times }$ indicates incorrect prediction. The text with a green background color indicates the ground truth actions. (b) Example of the text response used to derive the action planning answer.
• Figure 4: Ablation experiments on HMM creation based on Qwen2-VL.