Surgical AI Copilot: Energy-Based Fourier Gradient Low-Rank Adaptation for Surgical LLM Agent Reasoning and Planning

TL;DR

Surgical AI Copilot addresses the need for real-time, structured intraoperative reasoning in image-guided pituitary surgery by introducing a Planner-Worker LLM agent. It leverages the PitAgent dataset for domain-specific planning and proposes DEFT-GaLore, a deterministic energy-based Fourier Transform gradient low-rank projection, to enable efficient fine-tuning of open-source LLMs for surgical tasks. Across planning and visual-question-answering benchmarks, this approach outperforms existing PEFT methods while reducing training overhead, demonstrating potential for scalable, memory-efficient surgical decision support. The work highlights significant implications for real-time surgical decision-making, with future work aimed at generalizing to broader procedures and additional multimodal models.

Abstract

Image-guided surgery demands adaptive, real-time decision support, yet static AI models struggle with structured task planning and providing interactive guidance. Large language models (LLMs)-powered agents offer a promising solution by enabling dynamic task planning and predictive decision support. Despite recent advances, the absence of surgical agent datasets and robust parameter-efficient fine-tuning techniques limits the development of LLM agents capable of complex intraoperative reasoning. In this paper, we introduce Surgical AI Copilot, an LLM agent for image-guided pituitary surgery, capable of conversation, planning, and task execution in response to queries involving tasks such as MRI tumor segmentation, endoscope anatomy segmentation, overlaying preoperative imaging with intraoperative views, instrument tracking, and surgical visual question answering (VQA). To enable structured agent planning, we develop the PitAgent dataset, a surgical context-aware planning dataset covering surgical tasks like workflow analysis, instrument localization, anatomical segmentation, and query-based reasoning. Additionally, we propose DEFT-GaLore, a Deterministic Energy-based Fourier Transform (DEFT) gradient projection technique for efficient low-rank adaptation of recent LLMs (e.g., LLaMA 3.2, Qwen 2.5), enabling their use as surgical agent planners. We extensively validate our agent's performance and the proposed adaptation technique against other state-of-the-art low-rank adaptation methods on agent planning and prompt generation tasks, including a zero-shot surgical VQA benchmark, demonstrating the significant potential for truly efficient and scalable surgical LLM agents in real-time operative settings.

Figures (5)

• Figure 1: (a) denotes Query and Response samples of PitAgent dataset. (b) denotes how the Planner-Worker architecture employs the dataset to enable multi-step decision-making and real-time guidance for surgical scenarios.
• Figure 2: The Planner-Worker architecture of our Surgical LLM Agent. The Planner analyzes the surgeon's query and generates planning prompts. Based on these prompts, the Worker dynamically calls multiple promptable AI models such as segmentation, tracking, and MRI registration to collaboratively answer surgical queries.
• Figure 3: Example of constructing DEFT-GaLore projection gradient for LLaMA 3.2 model.
• Figure 4: Qualitative results analysis of agent planning. The responses are generated by Ground Truth, GaLore fine-tuned Planner and our DEFT-GaLore fine-tuned Planner.
• Figure 5: Qualitative result analysis of Worker on Planner-generated prompt vs GT prompt using publicly available zero-shot PitVQA++ model he2025pitvqa++.