SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability

TL;DR

This work argues that traditional sequence-based evaluations poorly reflect goal achievement in surgical planning by not accounting for hierarchical phase goals and procedural dependencies. It introduces a goal-satisfiability meta-evaluation benchmark built on expert surgical rules and a rule-based checker as a high-precision reference, revealing systematic misalignment of common metrics with true planning validity. Through progressive evaluation of Video-LLMs on MultiBypass140, the study shows that perception bottlenecks limit end-to-end planning, while under-constrained reasoning hinders scalable plan generation; explicit structural knowledge yields the most robust gains, especially for smaller models, whereas semantic guidance helps primarily in larger models when combined with structure. The findings advocate shifting from single-trajectory similarity to goal-satisfiability evaluation to drive clinically reliable surgical planning systems and highlight the need for scalable rule-based evaluation or robustHybrid knowledge integration in future work.

Abstract

Surgical planning integrates visual perception, long-horizon reasoning, and procedural knowledge, yet it remains unclear whether current evaluation protocols reliably assess vision-language models (VLMs) in safety-critical settings. Motivated by a goal-oriented view of surgical planning, we define planning correctness via phase-goal satisfiability, where plan validity is determined by expert-defined surgical rules. Based on this definition, we introduce a multicentric meta-evaluation benchmark with valid procedural variations and invalid plans containing order and content errors. Using this benchmark, we show that sequence similarity metrics systematically misjudge planning quality, penalizing valid plans while failing to identify invalid ones. We therefore adopt a rule-based goal-satisfiability metric as a high-precision meta-evaluation reference to assess Video-LLMs under progressively constrained settings, revealing failures due to perception errors and under-constrained reasoning. Structural knowledge consistently improves performance, whereas semantic guidance alone is unreliable and benefits larger models only when combined with structural constraints.

Figures (2)

• Figure 1: Comparison of sequence similarity metrics and rule-based checker metric.Top: Surgical procedures follow a hierarchical structure: each phase (e.g., P5) contains mandatory core steps (blue) and permissible generic steps (green). Bottom Left: Sequence similarity metrics compare predictions to a fixed reference, causing false negatives for valid clinical variations (Prediction A) and false positives for prohibited orderings (Prediction B). Bottom Right: The rule-based checker correctly distinguishes valid from invalid plans using surgical rules.(Section \ref{['rules']})
• Figure 2: Meta-evaluation and Evaluation Pipelines for Surgical Planning. Left: Rule-based benchmark defining goal-satisfiability via hierarchical phase-step relations and procedural constraints (dependencies and prohibitive orderings), separating valid and invalid step sequences. Right: Progressive evaluation of Video-LLMs, from end-to-end planning to planning with ground-truth steps and injected knowledge.