Anatomy Might Be All You Need: Forecasting What to Do During Surgery

TL;DR

The study addresses forecasting the next surgical-tool action in endoscopic neurosurgery to provide fine-grained guidance. It introduces a pipeline that combines anatomy-aware detections with a transformer-based forecaster to predict future tool motion over multiple frames, trained with an $L_1$ loss and a cosine-direction term. Supervisory signals are generated via a YOLOv7 detector applied to endoscopic videos, enabling ground-truth-like training without explicit trajectory labels. Results on pituitary surgery videos show that incorporating anatomical context improves directional forecasting accuracy, suggesting potential for real-time guidance and enhanced surgical training.

Abstract

Surgical guidance can be delivered in various ways. In neurosurgery, spatial guidance and orientation are predominantly achieved through neuronavigation systems that reference pre-operative MRI scans. Recently, there has been growing interest in providing live guidance by analyzing video feeds from tools such as endoscopes. Existing approaches, including anatomy detection, orientation feedback, phase recognition, and visual question-answering, primarily focus on aiding surgeons in assessing the current surgical scene. This work aims to provide guidance on a finer scale, aiming to provide guidance by forecasting the trajectory of the surgical instrument, essentially addressing the question of what to do next. To address this task, we propose a model that not only leverages the historical locations of surgical instruments but also integrates anatomical features. Importantly, our work does not rely on explicit ground truth labels for instrument trajectories. Instead, the ground truth is generated by a detection model trained to detect both anatomical structures and instruments within surgical videos of a comprehensive dataset containing pituitary surgery videos. By analyzing the interaction between anatomy and instrument movements in these videos and forecasting future instrument movements, we show that anatomical features are a valuable asset in addressing this challenging task. To the best of our knowledge, this work is the first attempt to address this task for manually operated surgeries.

Figures (2)

• Figure 1: Overview of the proposed model pipeline. Frames extracted from a surgical video are processed through an object detection network, which identifies and localizes anatomical structures and surgical instruments across a sequence of frames. These detected sequences are then passed into an encoder to extract temporal features. Finally, the forecasting head predicts the bounding box changes of the instrument for the next $f$ frames.
• Figure 2: Qualitative comparison of the forecasting models predicting 8 frames. For visualization purposes, only the predicted future bounding box centers are displayed. The yellow crosses indicate predictions closest to the current frame, while the purple crosses represent predictions furthest into the future. Additionally, the anatomy detections for the model that takes these as input are visualized.