Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery

TL;DR

The study investigates how well DeepSeek-based vision-language models can reason about robotic-assisted surgery through three dialogue tasks on EndoVis18 and CholecT50: Single Phrase QA, Visual QA, and Detailed Description. By comparing GPT-4o with DeepSeek variants (Janus-Pro-7b, VL2, and V3), and using image-tokenization to feed V3, the authors reveal that DeepSeek-V3 can handle simple QA when given image tokens, but all models struggle with global surgical understanding and detailed, clinically meaningful reasoning. DeepSeek-VL2 offers some context-aware descriptions, while DeepSeek-Janus-Pro-7b tends to ignore instructions or provide terse responses; none achieve robust, deployment-ready performance. The results underscore the need for surgery-specific fine-tuning and high-quality vision-language surgical datasets, and point to prompts, domain-adaptive training, and efficiency techniques as avenues to improve practical applicability in clinical settings.

Abstract

The DeepSeek models have shown exceptional performance in general scene understanding, question-answering (QA), and text generation tasks, owing to their efficient training paradigm and strong reasoning capabilities. In this study, we investigate the dialogue capabilities of the DeepSeek model in robotic surgery scenarios, focusing on tasks such as Single Phrase QA, Visual QA, and Detailed Description. The Single Phrase QA tasks further include sub-tasks such as surgical instrument recognition, action understanding, and spatial position analysis. We conduct extensive evaluations using publicly available datasets, including EndoVis18 and CholecT50, along with their corresponding dialogue data. Our empirical study shows that, compared to existing general-purpose multimodal large language models, DeepSeek-VL2 performs better on complex understanding tasks in surgical scenes. Additionally, although DeepSeek-V3 is purely a language model, we find that when image tokens are directly inputted, the model demonstrates better performance on single-sentence QA tasks. However, overall, the DeepSeek models still fall short of meeting the clinical requirements for understanding surgical scenes. Under general prompts, DeepSeek models lack the ability to effectively analyze global surgical concepts and fail to provide detailed insights into surgical scenarios. Based on our observations, we argue that the DeepSeek models are not ready for vision-language tasks in surgical contexts without fine-tuning on surgery-specific datasets.

Figures (2)

• Figure 1: Qualitative examples of Single Phrase QA, Visual QA, and Detailed Description on the EndoVis18 dataset.
• Figure 2: Qualitative examples of Single Phrase QA, Visual QA, and Detailed Description on the EndoVis18 dataset.