TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models

TL;DR

TrajPred is proposed, a framework that encodes instrument trajectories to incorporate temporal motion cues and, conditioned on these trajectories, introduces a predictor module to generate visual semantic embeddings that better capture fine-grained action details that improves alignment between relevant visual and textual representations.

Abstract

Recognizing instruments' interactions with tissues is essential for building context-aware AI assistants in robotic surgery. Vision-language models (VLMs) have opened a new avenue for surgical perception and achieved better generalization on a wide range of tasks compared to conventional task-specific deep learning approaches. However, their performance on instrument--tissue interaction recognition remains limited, largely due to two challenges: (1) many models do not effectively leverage temporal information, and (2) alignment between vision and text often misses fine-grained action details. To address these issues, we propose TrajPred, a framework that encodes instrument trajectories to incorporate temporal motion cues and, conditioned on these trajectories, introduces a predictor module to generate visual semantic embeddings that better capture fine-grained action details. We further incorporate prompt tuning and a verb-rephrasing technique to enable smooth adaptation to the instrument--tissue interaction recognition task. Extensive experiments on the public laparoscopic benchmark, CholecT50, show that our method improves both Average Precision and Top-K accuracy. We also investigate whether visual embeddings of instrument--tissue interaction regions align better with the corresponding text by visualizing the cosine similarity between visual and textual embeddings. The visualization results indicate that the proposed method improves alignment between relevant visual and textual representations.

Figures (2)

• Figure 1: (a) Cosine similarity heatmaps between embeddings of image patches and the text (e.g., “grasper retract gallbladder”). Two representative contrastive learning-based VLMs (CLIP-ViT-L radford2021learning and SurgVLP yuan2023surgvlp) exhibit high similarity between background patches and the text, whereas TrajPred (ours) produces embeddings that concentrate on the Instrument-Tissue interaction region. (b) Comparison of the architectures of (i) popular contrastive learning-based surgical VLMs, (ii) VL-JEPAchen2025vl, and (iii) TrajPred.
• Figure 2: Comparison between frames of the actions (a)"Grasper grasp" and (b)"Hook dissect".