Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognition

Bokai Zhang, Mohammad Hasan Sarhan, Bharti Goel, Svetlana Petculescu, Amer Ghanem

TL;DR

The paper tackles automatic surgical phase recognition by leveraging temporal information across long surgical videos. It introduces SF-TMN, a SlowFast-inspired two-path temporal modeling network that enables both frame-level and segment-level temporal modeling, trained in a two-stage process with a ResNet50 feature extractor and backbone options MS-TCN or ASFormer. Empirical results show SF-TMN achieving state-of-the-art performance on Cholec80 (notably with the ASFormer backbone) and strong results on 50Salads, GTEA, and Breakfast, with ablation studies validating design choices such as segment length and refinement strategy. The approach offers a robust, flexible framework for surgical phase recognition with practical impacts for VBA systems, video tagging, and phase-aware navigation in surgical video libraries.

Abstract

Automatic surgical phase recognition is one of the key technologies to support Video-Based Assessment (VBA) systems for surgical education. Utilizing temporal information is crucial for surgical phase recognition, hence various recent approaches extract frame-level features to conduct full video temporal modeling. For better temporal modeling, we propose SlowFast Temporal Modeling Network (SF-TMN) for surgical phase recognition that can not only achieve frame-level full video temporal modeling but also achieve segment-level full video temporal modeling. We employ a feature extraction network, pre-trained on the target dataset, to extract features from video frames as the training data for SF-TMN. The Slow Path in SF-TMN utilizes all frame features for frame temporal modeling. The Fast Path in SF-TMN utilizes segment-level features summarized from frame features for segment temporal modeling. The proposed paradigm is flexible regarding the choice of temporal modeling networks. We explore MS-TCN and ASFormer models as temporal modeling networks and experiment with multiple combination strategies for Slow and Fast Paths. We evaluate SF-TMN on Cholec80 surgical phase recognition task and demonstrate that SF-TMN can achieve state-of-the-art results on all considered metrics. SF-TMN with ASFormer backbone outperforms the state-of-the-art Not End-to-End(TCN) method by 2.6% in accuracy and 7.4% in the Jaccard score. We also evaluate SF-TMN on action segmentation datasets including 50salads, GTEA, and Breakfast, and achieve state-of-the-art results. The improvement in the results shows that combining temporal information from both frame level and segment level by refining outputs with temporal refinement stages is beneficial for the temporal modeling of surgical phases.

SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognition

TL;DR

The paper tackles automatic surgical phase recognition by leveraging temporal information across long surgical videos. It introduces SF-TMN, a SlowFast-inspired two-path temporal modeling network that enables both frame-level and segment-level temporal modeling, trained in a two-stage process with a ResNet50 feature extractor and backbone options MS-TCN or ASFormer. Empirical results show SF-TMN achieving state-of-the-art performance on Cholec80 (notably with the ASFormer backbone) and strong results on 50Salads, GTEA, and Breakfast, with ablation studies validating design choices such as segment length and refinement strategy. The approach offers a robust, flexible framework for surgical phase recognition with practical impacts for VBA systems, video tagging, and phase-aware navigation in surgical video libraries.

Abstract

Automatic surgical phase recognition is one of the key technologies to support Video-Based Assessment (VBA) systems for surgical education. Utilizing temporal information is crucial for surgical phase recognition, hence various recent approaches extract frame-level features to conduct full video temporal modeling. For better temporal modeling, we propose SlowFast Temporal Modeling Network (SF-TMN) for surgical phase recognition that can not only achieve frame-level full video temporal modeling but also achieve segment-level full video temporal modeling. We employ a feature extraction network, pre-trained on the target dataset, to extract features from video frames as the training data for SF-TMN. The Slow Path in SF-TMN utilizes all frame features for frame temporal modeling. The Fast Path in SF-TMN utilizes segment-level features summarized from frame features for segment temporal modeling. The proposed paradigm is flexible regarding the choice of temporal modeling networks. We explore MS-TCN and ASFormer models as temporal modeling networks and experiment with multiple combination strategies for Slow and Fast Paths. We evaluate SF-TMN on Cholec80 surgical phase recognition task and demonstrate that SF-TMN can achieve state-of-the-art results on all considered metrics. SF-TMN with ASFormer backbone outperforms the state-of-the-art Not End-to-End(TCN) method by 2.6% in accuracy and 7.4% in the Jaccard score. We also evaluate SF-TMN on action segmentation datasets including 50salads, GTEA, and Breakfast, and achieve state-of-the-art results. The improvement in the results shows that combining temporal information from both frame level and segment level by refining outputs with temporal refinement stages is beneficial for the temporal modeling of surgical phases.
Paper Structure (21 sections, 4 equations, 5 figures, 7 tables)

This paper contains 21 sections, 4 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Feature extraction network
  4. Action segmentation network
  5. MS-TCN
  6. ASFormer
  7. SF-TMN
  8. Experiments
  9. Dataset
  10. Evaluation metrics
  11. Implementation details
  12. Results
  13. Effect of the number of stages in MS-TCN
  14. Effect of the number of decoders in ASFormer
  15. Effects of segment length
  16. ...and 6 more sections

Figures (5)

  • Figure 1: The overview of our method
  • Figure 2: Action segmentation network overview: (a) SF-TMN (b) MS-TCN (c) ASFormer (d) SF-TMN(MS-TCN) (e) SF-TMN(ASFormer)
  • Figure 3: Different designs for SF-TMN: (a) Slow Path temporal refinement utilizes the combined output (b) Fast Path temporal refinement utilizes the combined output (c) Neither Slow Path temporal refinement nor Fast Path temporal refinement utilizes the combined output (d) Both Slow Path temporal refinement and Fast Path temporal refinement utilize the combined output
  • Figure 4: Color-coded ribbon illustration for surgical phase recognition: (a) MS-TCN (b) SF-TMN(MS-TCN) (c) ASFormer (d) SF-TMN(ASFormer) (e) Ground Truth
  • Figure 5: Application examples for our surgical phase recognition model: (a) the surgical phase navigation bar (b) AI-based intelligent video search with keywords (c) Surgical phase time documentation and comparison.