Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI-Driven Evaluation of Surgical Skill via Action Recognition

Yan Meng, Daniel A. Donoho, Marcelle Altshuler, Omar Arnaout

TL;DR

The paper addresses the challenge of objective, fine-grained assessment of microsurgical skill from video by proposing an AI framework that combines TimeSformer-based action segmentation with YOLO-DeepSORT derived instrument kinematics. It introduces hierarchical temporal attention and variance-weighted spatial attention to achieve precise, action-level segmentation aligned with NOMAT rubrics, backed by a self-guided practice kit for data collection. The approach yields strong segmentation performance (frame-level accuracy around 87.7% before post-processing and 93.62% after; average skill classification about 76%), and demonstrates interpretable, task-specific feedback for five NOMAT aspects, supporting scalable, data-driven training in resource-limited settings. Overall, the framework offers a practical pathway to democratize high-quality surgical skill assessment and feedback, with potential deployment via cloud services to improve global surgical education.

Abstract

The development of effective training and evaluation strategies is critical. Conventional methods for assessing surgical proficiency typically rely on expert supervision, either through onsite observation or retrospective analysis of recorded procedures. However, these approaches are inherently subjective, susceptible to inter-rater variability, and require substantial time and effort from expert surgeons. These demands are often impractical in low- and middle-income countries, thereby limiting the scalability and consistency of such methods across training programs. To address these limitations, we propose a novel AI-driven framework for the automated assessment of microanastomosis performance. The system integrates a video transformer architecture based on TimeSformer, improved with hierarchical temporal attention and weighted spatial attention mechanisms, to achieve accurate action recognition within surgical videos. Fine-grained motion features are then extracted using a YOLO-based object detection and tracking method, allowing for detailed analysis of instrument kinematics. Performance is evaluated along five aspects of microanastomosis skill, including overall action execution, motion quality during procedure-critical actions, and general instrument handling. Experimental validation using a dataset of 58 expert-annotated videos demonstrates the effectiveness of the system, achieving 87.7% frame-level accuracy in action segmentation that increased to 93.62% with post-processing, and an average classification accuracy of 76% in replicating expert assessments across all skill aspects. These findings highlight the system's potential to provide objective, consistent, and interpretable feedback, thereby enabling more standardized, data-driven training and evaluation in surgical education.

AI-Driven Evaluation of Surgical Skill via Action Recognition

TL;DR

The paper addresses the challenge of objective, fine-grained assessment of microsurgical skill from video by proposing an AI framework that combines TimeSformer-based action segmentation with YOLO-DeepSORT derived instrument kinematics. It introduces hierarchical temporal attention and variance-weighted spatial attention to achieve precise, action-level segmentation aligned with NOMAT rubrics, backed by a self-guided practice kit for data collection. The approach yields strong segmentation performance (frame-level accuracy around 87.7% before post-processing and 93.62% after; average skill classification about 76%), and demonstrates interpretable, task-specific feedback for five NOMAT aspects, supporting scalable, data-driven training in resource-limited settings. Overall, the framework offers a practical pathway to democratize high-quality surgical skill assessment and feedback, with potential deployment via cloud services to improve global surgical education.

Abstract

The development of effective training and evaluation strategies is critical. Conventional methods for assessing surgical proficiency typically rely on expert supervision, either through onsite observation or retrospective analysis of recorded procedures. However, these approaches are inherently subjective, susceptible to inter-rater variability, and require substantial time and effort from expert surgeons. These demands are often impractical in low- and middle-income countries, thereby limiting the scalability and consistency of such methods across training programs. To address these limitations, we propose a novel AI-driven framework for the automated assessment of microanastomosis performance. The system integrates a video transformer architecture based on TimeSformer, improved with hierarchical temporal attention and weighted spatial attention mechanisms, to achieve accurate action recognition within surgical videos. Fine-grained motion features are then extracted using a YOLO-based object detection and tracking method, allowing for detailed analysis of instrument kinematics. Performance is evaluated along five aspects of microanastomosis skill, including overall action execution, motion quality during procedure-critical actions, and general instrument handling. Experimental validation using a dataset of 58 expert-annotated videos demonstrates the effectiveness of the system, achieving 87.7% frame-level accuracy in action segmentation that increased to 93.62% with post-processing, and an average classification accuracy of 76% in replicating expert assessments across all skill aspects. These findings highlight the system's potential to provide objective, consistent, and interpretable feedback, thereby enabling more standardized, data-driven training and evaluation in surgical education.
Paper Structure (21 sections, 5 equations, 5 figures, 2 tables)

This paper contains 21 sections, 5 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video-Based Surgical Skill Assessment.
  4. Temporal Action Segmentation.
  5. Methodology
  6. Self-guided Microanastomosis Practice Kit
  7. Transformer-based Action Segmentation
  8. Hierarchical Temporal Attention
  9. Global Temporal Attention.
  10. Local Temporal Attention.
  11. Variance-Weighted Spatial Self-Attention
  12. Kinematic Feature Extraction
  13. Microanastomosis Skill Classification
  14. Experiments
  15. Data Collection
  16. ...and 6 more sections

Figures (5)

  • Figure 1: An overview of the transformer-based microanastomosis skill assessment framework.
  • Figure 2: Microvascular practice cards and the suturing stitches illustration.
  • Figure 3: The transformer architecture for microanastomosis action segmentation.
  • Figure 4: Comparison of action segmentation results across different methods against the ground truth.
  • Figure 5: YOLO-based instrument tip tracking.