CurConMix+: A Unified Spatio-Temporal Framework for Hierarchical Surgical Workflow Understanding

TL;DR

CurConMix+ introduces a unified spatio-temporal framework for hierarchical surgical workflow understanding by coupling curriculum-guided contrastive learning with a Multi-Resolution Temporal Transformer (MRTT). The approach jointly tackles class imbalance, subtle visual variation, and compositional dependencies among triplet components, and extends to a new benchmark, LLS48, for laparoscopic left lateral sectionectomy. Empirical results on CholecT45 and LLS48 show state-of-the-art performance in fine-grained action triplet recognition and strong transfer to higher-level phase/step/task tasks, with ablations validating the contribution of each component. The work offers a reproducible foundation for hierarchy-aware, interpretable surgical understanding, with public release of code and dataset.

Abstract

Surgical action triplet recognition aims to understand fine-grained surgical behaviors by modeling the interactions among instruments, actions, and anatomical targets. Despite its clinical importance for workflow analysis and skill assessment, progress has been hindered by severe class imbalance, subtle visual variations, and the semantic interdependence among triplet components. Existing approaches often address only a subset of these challenges rather than tackling them jointly, which limits their ability to form a holistic understanding. This study builds upon CurConMix, a spatial representation framework. At its core, a curriculum-guided contrastive learning strategy learns discriminative and progressively correlated features, further enhanced by structured hard-pair sampling and feature-level mixup. Its temporal extension, CurConMix+, integrates a Multi-Resolution Temporal Transformer (MRTT) that achieves robust, context-aware understanding by adaptively fusing multi-scale temporal features and dynamically balancing spatio-temporal cues. Furthermore, we introduce LLS48, a new, hierarchically annotated benchmark for complex laparoscopic left lateral sectionectomy, providing step-, task-, and action-level annotations. Extensive experiments on CholecT45 and LLS48 demonstrate that CurConMix+ not only outperforms state-of-the-art approaches in triplet recognition, but also exhibits strong cross-level generalization, as its fine-grained features effectively transfer to higher-level phase and step recognition tasks. Together, the framework and dataset provide a unified foundation for hierarchy-aware, reproducible, and interpretable surgical workflow understanding. The code and dataset will be publicly released on GitHub to facilitate reproducibility and further research.

Figures (3)

• Figure 1: Overview of CurConMix+. The framework comprises two core modules: (1) the CurConMix spatial backbone and (2) the MRTT temporal module. CurConMix employs curriculum-guided contrastive learning with hard-pair sampling and feature-level mixup to learn discriminative spatial features. The pretrained backbone is then fine-tuned via a self-distillation scheme incorporating input mixup to mitigate class imbalance. Subsequently, the MRTT module captures temporal dynamics through parallel temporal pathways, whose outputs are adaptively fused using $\gamma_k$ and dynamically weighted with spatial cues via $\beta$. The complete CurConMix+ pipeline integrates these spatial and temporal components to achieve robust spatio-temporal triplet prediction.
• Figure 2: Grad-CAM visualization of CurConMix on CholecT45 and LLS48.
• Figure 3: Comparison of minority-class performance between TERL and CurConMix+ on CholecT45 and LLS48.