State-Change Learning for Prediction of Future Events in Endoscopic Videos

TL;DR

This paper reframes surgical future prediction as state-change learning, where transitions between adjacent video clips are categorized into four labels. It introduces SurgFUTR, a teacher–student framework that uses Sinkhorn clustering to form state vectors, a state graph to model centroid interactions, and an ActDyn module to predict future centroid transitions, all facilitating state-change pretraining. A new benchmark, SFPBench, aggregates five tasks across short- and long-term horizons and multiple procedures, enabling comprehensive evaluation and cross-procedure transfer. Across four datasets and three procedures, state-change pretraining yields consistent improvements over baselines, with the full SurgFUTR-TS variant delivering the strongest gains on downstream RSD, phase/step transitions, cystic triplets, and cholec events, validating the approach's generalization and practical impact for predictive surgical AI.

Abstract

Surgical future prediction, driven by real-time AI analysis of surgical video, is critical for operating room safety and efficiency. It provides actionable insights into upcoming events, their timing, and risks-enabling better resource allocation, timely instrument readiness, and early warnings for complications (e.g., bleeding, bile duct injury). Despite this need, current surgical AI research focuses on understanding what is happening rather than predicting future events. Existing methods target specific tasks in isolation, lacking unified approaches that span both short-term (action triplets, events) and long-term horizons (remaining surgery duration, phase transitions). These methods rely on coarse-grained supervision while fine-grained surgical action triplets and steps remain underexplored. Furthermore, methods based only on future feature prediction struggle to generalize across different surgical contexts and procedures. We address these limits by reframing surgical future prediction as state-change learning. Rather than forecasting raw observations, our approach classifies state transitions between current and future timesteps. We introduce SurgFUTR, implementing this through a teacher-student architecture. Video clips are compressed into state representations via Sinkhorn-Knopp clustering; the teacher network learns from both current and future clips, while the student network predicts future states from current videos alone, guided by our Action Dynamics (ActDyn) module. We establish SFPBench with five prediction tasks spanning short-term (triplets, events) and long-term (remaining surgery duration, phase and step transitions) horizons. Experiments across four datasets and three procedures show consistent improvements. Cross-procedure transfer validates generalizability.

Figures (13)

• Figure 1: Traditional anticipation models (a) predict raw future features (1) or use phase/instrument anticipation heads (2) without considering semantic state transitions. Our approach (b) reformulates anticipation as state-change classification, categorizing transitions between consecutive clips into four labels: continuity, discontinuity, onset, or background. (c) illustrates this for CholecT50 verbs: each half-circle can be green (class present) or red (class absent), with the left representing time $t$ and right representing $t+\delta$. The combination yields four state-change labels: (green, green) = continuity, (green, red) = discontinuity, (red, green) = onset, (red, red) = background. This state-transition learning builds future-aware representations that enhance downstream prediction performance.
• Figure 2: SFPBench: We introduce a comprehensive benchmark for evaluating state-change pretraining across multiple surgical procedures and prediction tasks. The benchmark encompasses four datasets spanning three laparoscopic surgical procedures (cholecystectomy, robotic-assisted radical prostatectomy, gastric bypass) and includes both long-term forecasting tasks (remaining surgery duration, phase/step transitions) and short-term anticipation tasks (cystic-structure related action triplets, surgical events). Cross-procedure transfer evaluation from cholecystectomy to gastric bypass validates generalization capabilities. See Section \ref{['task_desc']} for details.
• Figure 3: SurgFUTR: a teacher-student framework comprising: (a) a video encoder for spatio-temporal feature extraction; (b) a state encoder using Sinkhorn-Knopp clustering to generate centroid-based state representations; (c) a state graph enabling cross-cluster information exchange through message passing; (d) an Action Dynamics (ActDyn) module predicting transitions from current-clip to future-clip centroids; and (e) a state decoder mapping states to per-class embeddings for state-change classification.
• Figure 4: SurgFUTR-Lite: A single-stage future feature prediction model. The main encoder processes a clip at time $t$; the target encoder processes a clip at $t+\delta$, with the target backbone as an exponential moving average of the main encoder's. Spatially pooled features $\mathbf{F}_{t}$ are used for future feature prediction to output $\mathbf{\hat{F}}_{t+\delta}$, and are trained to match $\mathbf{F}_{t+\delta}$ via $\mathcal{L}_{FUTR}$. The concatenated features $[\mathbf{F}_{t},\mathbf{\hat{F}}_{t+\delta}]$ also drive a state-change head optimized with $\mathcal{L}_{SC}$.
• Figure 5: Action dynamics modeling with centroid transition prediction