General surgery vision transformer: A video pre-trained foundation model for general surgery

TL;DR

This work tackles the lack of openly accessible surgical data and foundation models by releasing GenSurgery, the largest public corpus of general surgery videos (680 hours, ~70M frames across 28 procedures), and introducing GSViT, a lightweight vision transformer pretrained via next-frame video prediction for real-time surgical use. GSViT combines a memory-efficient sandwich architecture with Cascaded Group Attention and an asymmetric decoder to learn spatio-temporal priors from surgical videos, and is accompanied by ten procedure-specific fine-tuned variants. In Cholec80-based evaluation, GSViT achieves strong single-frame phase-detection performance (e.g., 86.3% accuracy) while offering substantially lower parameter counts and real-time throughput (~12 ms per image on a RTX A5500) compared with some baselines. By providing open-source data, models, and code, the work advances scalable, real-time surgical AI and lays groundwork for broader, multimodal surgical foundation models.

Abstract

The absence of openly accessible data and specialized foundation models is a major barrier for computational research in surgery. Toward this, (i) we open-source the largest dataset of general surgery videos to-date, consisting of 680 hours of surgical videos, including data from robotic and laparoscopic techniques across 28 procedures; (ii) we propose a technique for video pre-training a general surgery vision transformer (GSViT) on surgical videos based on forward video prediction that can run in real-time for surgical applications, toward which we open-source the code and weights of GSViT; (iii) we also release code and weights for procedure-specific fine-tuned versions of GSViT across 10 procedures; (iv) we demonstrate the performance of GSViT on the Cholec80 phase annotation task, displaying improved performance over state-of-the-art single frame predictors.

Figures (3)

• Figure 1: Graphical depiction of the training process for GSViT. Video prediction-based pre-training with asymmetric decoder head for video frame reconstruction (right). Demonstration of application-specific fine-tuning using GSViT with frozen weights and learned classification head (left).
• Figure 2: Architecture diagram of GSViT.
• Figure 3: GenSurgery.