Model selection and real-time skill assessment for suturing in robotic surgery

TL;DR

The paper addresses real-time, continuous surgical skill assessment in robot-assisted suturing by predicting OSATS scores from synchronized kinematic and video data. It systematically evaluates seven deep models across unimodal and multimodal architectures, using windowed processing and three cross-validation schemes to probe generalization by surgeon skill level. The findings demonstrate that multimodal fusion yields more stable, accurate real-time predictions than unimodal baselines, with expert-level training data markedly improving cross-participant generalization. The work highlights the practical potential of real-time feedback in robotic surgery and outlines avenues for richer annotations and larger, more balanced datasets to further enhance robustness and clinical usefulness.

Abstract

Automated feedback systems have the potential to provide objective skill assessment for training and evaluation in robot-assisted surgery. In this study, we examine methods to achieve real-time prediction of surgical skill level in real-time based on Objective Structured Assessment of Technical Skills (OSATS) scores. Using data acquired from the da Vinci Surgical System, we carry out three main analyses, focusing on model design, their real-time performance, and their skill-level-based cross-validation training. For the model design, we evaluate the effectiveness of multimodal deep learning models for predicting surgical skill levels using synchronized kinematic and vision data. Our models include separate unimodal baselines and fusion architectures that integrate features from both modalities and are evaluated using mean Spearman's correlation coefficients, demonstrating that the fusion model consistently outperforms unimodal models for real-time predictions. For the real-time performance, we observe the prediction's trend over time and highlight correlation with the surgeon's gestures. For the skill-level-based cross-validation, we separately trained models on surgeons with different skill levels, which showed that high-skill demonstrations allow for better performance than those trained on low-skilled ones and generalize well to similarly skilled participants. Our findings show that multimodal learning allows more stable fine-grained evaluation of surgical performance and highlights the value of expert-level training data for model generalization.

Figures (4)

• Figure 1: Suturing example in the JIGSAWS dataset. As the needle is pulled out, the tissue is visibly torn, which affects the skill assessment score "Respect for Tissue".
• Figure 2: Real-time prediction of OSATS scores for two representative sample subjects. Each subplot corresponds to a different OSATS$^{1\text{-}3}$ component, where the x-axis denotes time and the y-axis reflects the prediction scores. The lines represent the ground truth label, and the predictions from the kinematic-only-based model, vision-based-only model, and the fusion model. The background is segmented with colored bands indicating the active gesture label $G$ as provided by JIGSAWS, providing contextual cues from the temporal evolution of surgical gestures.
• Figure 3: Results from LOSI cross validation using kinematic-only model LSTM-K with testing on OSATS$^{1\text{-}3}=[n,n,n]$.
• Figure 4: Results from LOSI cross validation using vision-only model CNN-LSTM-V with testing on OSATS$^{1\text{-}3}=[n,n,n]$.