SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognition

TL;DR

This paper develops SurgPETL, a parameter-efficient transfer learning framework for surgical phase recognition, and introduces the Adaptive Spatiotemporal Representation Modulation (ASRM) module, which outperforms both parameter-efficient alternatives and state-of-the-art surgical phase recognition methods while maintaining parameter efficiency and minimizing overhead.

Abstract

Capitalizing on image-level pre-trained models for various downstream tasks has recently emerged with promising performance. However, the paradigm of "image pre-training followed by video fine-tuning" for high-dimensional video data inevitably poses significant performance bottlenecks. Furthermore, in the medical domain, many surgical video tasks encounter additional challenges posed by the limited availability of video data and the necessity for comprehensive spatial-temporal modeling. Recently, Parameter-Efficient Image-to-Video Transfer Learning has emerged as an efficient and effective paradigm for video action recognition tasks, which employs image-level pre-trained models with promising feature transferability and involves cross-modality temporal modeling with minimal fine-tuning. Nevertheless, the effectiveness and generalizability of this paradigm within intricate surgical domain remain unexplored. In this paper, we delve into a novel problem of efficiently adapting image-level pre-trained models to specialize in fine-grained surgical phase recognition, termed as Parameter-Efficient Image-to-Surgical-Video Transfer Learning. Firstly, we develop a parameter-efficient transfer learning benchmark SurgPETL for surgical phase recognition, and conduct extensive experiments with three advanced methods based on ViTs of two distinct scales pre-trained on five large-scale natural and medical datasets. Then, we introduce the Spatial-Temporal Adaptation module, integrating a standard spatial adapter with a novel temporal adapter to capture detailed spatial features and establish connections across temporal sequences for robust spatial-temporal modeling. Extensive experiments on three challenging datasets spanning various surgical procedures demonstrate the effectiveness of SurgPETL with STA.

Figures (6)

• Figure 1: Illustration of two distinct paradigms for surgical phase recognition. Left: two-stage full fine-tuning paradigm; Right: end-to-end parameter-efficient image-to-surgical-video transfer learning paradigm.
• Figure 2: Illustration of the investigation with two distinct phases for parameter-efficient image-to-surgical-video transfer learning.
• Figure 3: Illustration of the selected PEIVTL method benchmarks.
• Figure 4: Overview of SurgPETL-STA. (a): SurgPETL-STA block consists of one Spatial-Temporal Adaption (STA) module and two adapters for temporal, spatial and joint adaptations. Each STA contains a spatial adapter, a novel temporal adapter and a spatial-temporal router. (b): Feature Re-embedding (FR) mechanism of the temporal adapter is utilized to fully utilize the potential temporal feature space for enhanced representation.
• Figure 5: Overall comparison of SurgPETL based on three state-of-the-art methods, employing ViT-B and ViT-L pre-trained on five distinct large-scale datasets. BOLD indicates the best performance achieved by each method with varying pre-trained parameters.