Evaluating Large Vision-language Models for Surgical Tool Detection

TL;DR

The paper tackles the challenge of holistic surgical scene understanding by evaluating three large vision-language models (Qwen2.5-7B, LLaVA1.5-7B, InternVL3.5-8B) on surgical instrument detection using the GraSP dataset under zero-shot and Rank-8 LoRA fine-tuning. It demonstrates that Qwen2.5 provides the strongest detection performance across both deployment modes, with better zero-shot generalization than the open-set baseline Grounding DINO, though GDINO often offers superior localization. Fine-tuning improves performance for all models, revealing complementary strengths: Qwen excels at instrument classification while GDINO yields more precise localization. The findings support the potential of general-purpose surgical VLMs to contribute to broader workflow understanding and highlight avenues for extending these models to phase, action, and step recognition in surgical AI systems.

Abstract

Surgery is a highly complex process, and artificial intelligence has emerged as a transformative force in supporting surgical guidance and decision-making. However, the unimodal nature of most current AI systems limits their ability to achieve a holistic understanding of surgical workflows. This highlights the need for general-purpose surgical AI systems capable of comprehensively modeling the interrelated components of surgical scenes. Recent advances in large vision-language models that integrate multimodal data processing offer strong potential for modeling surgical tasks and providing human-like scene reasoning and understanding. Despite their promise, systematic investigations of VLMs in surgical applications remain limited. In this study, we evaluate the effectiveness of large VLMs for the fundamental surgical vision task of detecting surgical tools. Specifically, we investigate three state-of-the-art VLMs, Qwen2.5, LLaVA1.5, and InternVL3.5, on the GraSP robotic surgery dataset under both zero-shot and parameter-efficient LoRA fine-tuning settings. Our results demonstrate that Qwen2.5 consistently achieves superior detection performance in both configurations among the evaluated VLMs. Furthermore, compared with the open-set detection baseline Grounding DINO, Qwen2.5 exhibits stronger zero-shot generalization and comparable fine-tuned performance. Notably, Qwen2.5 shows superior instrument recognition, while Grounding DINO demonstrates stronger localization.

Figures (2)

• Figure 1: Illustration of error types. Green boxes indicate ground truth, and orange boxes indicate predictions corresponding to a specific error category. Shorthand instrument names are shown, along with the Intersection over Union (IoU) between the prediction and the ground truth box.
• Figure 2: Qualitative comparison of surgical instrument detection across vision language models. Predictions from Qwen, LLaVA, and InternVL are shown alongside the GDINO baseline under both zero-shot (ZS) and fine-tuned (FT) settings. The green bounding box indicates ground truth, the blue bounding box indicates a correct detection, and the orange bounding box indicates an incorrect detection. Bounding boxes are annotated with abbreviated instrument class labels and, where applicable, their Intersection over Union (IoU) scores relative to the ground-truth (GT) annotations.