Expert Consensus-based Video-Based Assessment Tool for Workflow Analysis in Minimally Invasive Colorectal Surgery: Development and Validation of ColoWorkflow

TL;DR

ColoWorkflow introduces the first consensus-derived, procedure-agnostic VBA tool for comprehensive workflow analysis in minimally invasive colorectal surgery. It was developed via a three-round Delphi with international experts to yield 10 phases and 34 steps, validated on multicenter MIS CRS videos, and demonstrated feasible applicability with moderate inter-rater reliability ($κ$≈0.71 for phases and $κ$≈0.66 for steps). This framework supports standardized workflow benchmarking, guides training and quality improvement, and serves as a foundation for AI-driven workflow recognition. The study acknowledges limitations in expert representation and labeling variability, but positions ColoWorkflow as a scalable, extensible platform for cross-center comparisons and data-informed surgical education.

Abstract

Minimally invasive colorectal surgery is characterized by procedural variability, a difficult learning curve, and complications that impact quality and outcomes. Video-based assessment (VBA) offers an opportunity to generate data-driven insights to reduce variability, optimize training, and improve surgical performance. However, existing tools for workflow analysis remain difficult to standardize and implement. This study aims to develop and validate a VBA tool for workflow analysis across minimally invasive colorectal procedures. A Delphi process was conducted to achieve consensus on generalizable workflow descriptors. The resulting framework informed the development of a new VBA tool, ColoWorkflow. Independent raters then applied ColoWorkflow to a multicentre video dataset of laparoscopic and robotic colorectal surgery (CRS). Applicability and inter-rater reliability were evaluated. Consensus was achieved for 10 procedure-agnostic phases and 34 procedure-specific steps describing CRS workflows. ColoWorkflow was developed and applied to 54 colorectal operative videos (left and right hemicolectomies, sigmoid and rectosigmoid resections, and total proctocolectomies) from five centres. The tool demonstrated broad applicability, with all but one label utilized. Inter-rater reliability was moderate, with mean Cohen's K of 0.71 for phases and 0.66 for steps. Most discrepancies arose at phase transitions and step boundary definitions. ColoWorkflow is the first consensus-based, validated VBA tool for comprehensive workflow analysis in minimally invasive CRS. It establishes a reproducible framework for video-based performance assessment, enabling benchmarking across institutions and supporting the development of artificial intelligence-driven workflow recognition. Its adoption may standardize training, accelerate competency acquisition, and advance data-informed surgical quality improvement.

Figures (4)

• Figure 1: Results of the modified Delphi process. The numbers for phases and steps represent both the inclusion of phases/steps and their corresponding descriptions as separate items for consensus.
• Figure 1: Phases and steps in the video-based assessment (VBA) tool ranked by number of expert suggestions given for each phase/step naming and description. All phases shown in (a) (above); Top 15 steps shown in (b) (below). Comments mostly revolved around alternate approaches, inclusion of anatomical landmarks, and clarity in wording.
• Figure 2: Representation of ColoWorkflow. Common phases are shown in dark blue, steps shared across left- and right-sided procedures in light blue, steps specific to left-sided procedures (left hemicolectomy, rectal resection, sigmoid resection) in yellow, and steps specific to right-sided procedures (right hemicolectomy) in orange.
• Figure 2: Inter-annotator differences between the annotators and distribution across the labels. The plots show the Cohen’s $\kappa$ metric and its variability for each label, indicating that some labels have high agreement with low variation (e.g., step 1), while others may have low agreement and high variability (e.g., step 20). Phases shown in (a) (above); Steps in (b) (below). Phase and step names correspond to the numbering given in eTables \ref{['etab:3']} and \ref{['etab:4']}.