Bridging Vision and Language for Robust Context-Aware Surgical Point Tracking: The VL-SurgPT Dataset and Benchmark

TL;DR

VL-SurgPT introduces the first large-scale multimodal dataset that pairs surgical point trajectories (tissue and instrument) with textual descriptions of point status, enabling context-aware tracking in challenging in vivo conditions. Building TG-SurgPT on a Track-On–style framework, the authors fuse visual features with semantic text via cross-modal attention and an attributes-prediction head, and optimize with a loss that jointly accounts for spatial accuracy, trajectory smoothness, and text classification. Empirical results show TG-SurgPT outperforming eight strong vision-only baselines across tissue and instrument tracking while operating at near real-time speeds, with notable robustness gains under smoke, occlusion, and deformation. This work demonstrates the practical value of integrating semantic context into surgical tracking, paving the way for more reliable, interpretable, and context-aware computer-assisted intervention systems.

Abstract

Accurate point tracking in surgical environments remains challenging due to complex visual conditions, including smoke occlusion, specular reflections, and tissue deformation. While existing surgical tracking datasets provide coordinate information, they lack the semantic context necessary to understand tracking failure mechanisms. We introduce VL-SurgPT, the first large-scale multimodal dataset that bridges visual tracking with textual descriptions of point status in surgical scenes. The dataset comprises 908 in vivo video clips, including 754 for tissue tracking (17,171 annotated points across five challenging scenarios) and 154 for instrument tracking (covering seven instrument types with detailed keypoint annotations). We establish comprehensive benchmarks using eight state-of-the-art tracking methods and propose TG-SurgPT, a text-guided tracking approach that leverages semantic descriptions to improve robustness in visually challenging conditions. Experimental results demonstrate that incorporating point status information significantly improves tracking accuracy and reliability, particularly in adverse visual scenarios where conventional vision-only methods struggle. By bridging visual and linguistic modalities, VL-SurgPT enables the development of context-aware tracking systems crucial for advancing computer-assisted surgery applications that can maintain performance even under challenging intraoperative conditions.

Figures (10)

• Figure 1: Overview of our Vision-Language Surgical Point Tracking (VL-SurgPT) dataset, a large-scale multimodal dataset containing visual and textual annotations for tissue and instrument points across diverse challenging scenarios.
• Figure 2: Data collection and annotation workflow for VL-SurgPT. (A) In vivo surgical setup using the da Vinci Xi system. (B) Ground truth acquisition using Indocyanine Green (ICG) fluorescent markers under UV illumination. (C-D) Annotation interface for point tracking and semantic labeling at 1 fps. (E) Coverage of 7 types of surgical instruments, 9 distinct visual status descriptions, and 5 representative challenging scenarios across our dataset.
• Figure 3: Overview of our Text-guided Surgical Point Tracking (TG-SurgPT). The method builds upon Track-On Aydemir2025trackon by integrating visual features with semantic text descriptions through cross-modal attention.
• Figure 4: Qualitative comparison of tracking performance of the finetuned Track-On Aydemir2025trackon and our TG-SurgPT across sequential frames (30, 60, 90, 120) for both instrument (top row) and tissue (bottom row) point tracking.
• Figure 5: Scenario-specific comparison ($<\delta^{x}_{avg}$, higher is better) across Tissue Deformation (TD), Instrument Occlusion (IO), Camera Jitter (CJ), Surface Reflection (SR), and Cauterization Smoke (CS). Box height indicates the mean performance of all methods.