Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement

TL;DR

This work tackles robust surgical workflow recognition in RMIS by fusing vision and kinematic data through a graph-based framework. The GRAD model combines Multimodal Disentanglement Graph Network (MDGNet) for intra- and inter-modal feature learning, Vision-Kinematic Adversarial (VKA) training to align modality distributions, and a Contextual Calibrated Decoder to stabilize predictions under corruption and domain shifts, all validated on MISAW and CUHK-MRG datasets. Comprehensive ablations demonstrate the contributions of visual disentanglement (spatial, wavelet, Fourier), graph attention fusion, and calibrated loss, with GRAD achieving state-of-the-art accuracy and robustness against 18 corruptions across five severities. The results highlight GRAD’s potential to improve automated workflow understanding in real-world surgical settings, enhancing automation, training, and patient safety.

Abstract

Surgical workflow recognition is vital for automating tasks, supporting decision-making, and training novice surgeons, ultimately improving patient safety and standardizing procedures. However, data corruption can lead to performance degradation due to issues like occlusion from bleeding or smoke in surgical scenes and problems with data storage and transmission. In this case, we explore a robust graph-based multimodal approach to integrating vision and kinematic data to enhance accuracy and reliability. Vision data captures dynamic surgical scenes, while kinematic data provides precise movement information, overcoming limitations of visual recognition under adverse conditions. We propose a multimodal Graph Representation network with Adversarial feature Disentanglement (GRAD) for robust surgical workflow recognition in challenging scenarios with domain shifts or corrupted data. Specifically, we introduce a Multimodal Disentanglement Graph Network that captures fine-grained visual information while explicitly modeling the complex relationships between vision and kinematic embeddings through graph-based message modeling. To align feature spaces across modalities, we propose a Vision-Kinematic Adversarial framework that leverages adversarial training to reduce modality gaps and improve feature consistency. Furthermore, we design a Contextual Calibrated Decoder, incorporating temporal and contextual priors to enhance robustness against domain shifts and corrupted data. Extensive comparative and ablation experiments demonstrate the effectiveness of our model and proposed modules. Moreover, our robustness experiments show that our method effectively handles data corruption during storage and transmission, exhibiting excellent stability and robustness. Our approach aims to advance automated surgical workflow recognition, addressing the complexities and dynamism inherent in surgical procedures.

Figures (7)

• Figure 1: Comparison between the vision-based workflow recognition framework and the multimodal workflow recognition framework.
• Figure 2: The overall architecture of our GRAD framework, consisting of the Multimodal Disentanglement Graph Network (Visual Representation Disentanglement module, the Kinematic Temporal Representation Extraction module, the Multimodal Graph Learning module), the Vision-Kinematic Adversarial Training, and the Calibrated Prediction Decoder. Our GRAD framework extracts visual and kinematic features through intra-modal feature mining. It uses adversarial learning to approximate the representation distributions of both. The graph network facilitates the interaction between the two modalities, and network calibration ensures robust prediction results.
• Figure 3: Visualization of GRAD compared to MRG-Net (Vis+Kin), Trans-SVNet (Vis), and RL-TCN (Kin) model on MISAW dataset. The different color bands represent each category, which are needle holding, suture making, suture handling, $1^ \circ$ knot, $2^ \circ$ knot, and $3^ \circ$ knot (with order).
• Figure 4: Figure of CUHK-MRG Dataset illustrates five different gesture steps, which are idle (no action performed), reach for peg (with left hand), lift peg (with left hand), exchange (transfer the peg to right hand), place peg (with right hand).
• Figure 5: An example of corrupted data visualization for robustness assessment, featuring four different types ranging from noise to digital damage.