Adaptive transfer learning for surgical tool presence detection in laparoscopic videos through gradual freezing fine-tuning

TL;DR

The paper tackles tool presence detection in laparoscopic videos under data-scarce conditions and domain shift, proposing Gradual Freezing (GFz), an adaptive fine-tuning method with two phases: Pre-conditioning (linear probing) and Gradual Freezing guided by Relative Gradient Norms to selectively freeze layers in a single training pass. Applied to ImageNet-pretrained ResNet-50 and DenseNet-121 on the Cholec80 dataset, GFz achieves state-of-the-art $mAP$ around 96% and demonstrates strong generalization to the CATARACTS dataset, with notable training efficiency gains. The work includes extensive ablations on hyperparameters (e.g., $oldsymbol{\ \epsilon_{cond}}$, $oldsymbol{\epsilon_{step}}$, and freezing fraction) and shows GFz can outperform conventional fine-tuning and several alternative strategies, while preserving inference speed. Overall, GFz provides a robust, efficient transfer-learning framework for surgical image analysis that can extend to other domains with substantial domain shifts.

Abstract

Minimally invasive surgery can benefit significantly from automated surgical tool detection, enabling advanced analysis and assistance. However, the limited availability of annotated data in surgical settings poses a challenge for training robust deep learning models. This paper introduces a novel staged adaptive fine-tuning approach consisting of two steps: a linear probing stage to condition additional classification layers on a pre-trained CNN-based architecture and a gradual freezing stage to dynamically reduce the fine-tunable layers, aiming to regulate adaptation to the surgical domain. This strategy reduces network complexity and improves efficiency, requiring only a single training loop and eliminating the need for multiple iterations. We validated our method on the Cholec80 dataset, employing CNN architectures (ResNet-50 and DenseNet-121) pre-trained on ImageNet for detecting surgical tools in cholecystectomy endoscopic videos. Our results demonstrate that our method improves detection performance compared to existing approaches and established fine-tuning techniques, achieving a mean average precision (mAP) of 96.4%. To assess its broader applicability, the generalizability of the fine-tuning strategy was further confirmed on the CATARACTS dataset, a distinct domain of minimally invasive ophthalmic surgery. These findings suggest that gradual freezing fine-tuning is a promising technique for improving tool presence detection in diverse surgical procedures and may have broader applications in general image classification tasks.

Figures (11)

• Figure 1: Overview of the proposed gradual freezing fine-tuning strategy for adaptive transfer learning. a. Pre-conditioning: All layers except the last classifier layer are frozen with pre-trained parameters and trained for $\epsilon_{\text{cond}}$ epochs, allowing the new, randomly initialized layer to adapt to the new task (Linear probing). b. Gradual freezing: All layers are unfrozen and layers with low importance, based on their relative gradient norm (RGN), are gradually frozen at each subsequent freezing step (every $\epsilon_{\text{step}}$ epochs).
• Figure 2: Sample images of labeled surgical tools in the Cholec80 dataset.
• Figure 3: Mean Average Precision (mAP) for all tools under the six implementations of the proposed fine-tuning strategy.
• Figure 4: Precision-Recall curves for the seven surgical tools using the L40 and L30 splits.
• Figure 5: Top: Number of layers trained across training epochs. Bottom: Heatmap representing the number of times each layer is trained across model layers.