Distilling Expert Surgical Knowledge: How to train local surgical VLMs for anatomy explanation in Complete Mesocolic Excision

TL;DR

The paper tackles the need for domain-aware surgical understanding while preserving patient privacy by distilling knowledge from cloud LLMs into a local CME-focused VLM. It introduces a dataset generation pipeline that uses textual context and binary segmentation masks rather than real images, followed by supervised fine-tuning and direct preference optimization. Automated metrics and expert evaluations show substantial gains from SFT over a base model, while DPO yields only marginal improvements, suggesting the expert-supervised data already aligns well with clinical practice. The approach demonstrates a practical, privacy-preserving route to deploy CME-aware VLMs for surgical explanation and education.

Abstract

Recently, Vision Large Language Models (VLMs) have demonstrated high potential in computer-aided diagnosis and decision-support. However, current VLMs show deficits in domain specific surgical scene understanding, such as identifying and explaining anatomical landmarks during Complete Mesocolic Excision. Additionally, there is a need for locally deployable models to avoid patient data leakage to large VLMs, hosted outside the clinic. We propose a privacy-preserving framework to distill knowledge from large, general-purpose LLMs into an efficient, local VLM. We generate an expert-supervised dataset by prompting a teacher LLM without sensitive images, using only textual context and binary segmentation masks for spatial information. This dataset is used for Supervised Fine-Tuning (SFT) and subsequent Direct Preference Optimization (DPO) of the locally deployable VLM. Our evaluation confirms that finetuning VLMs with our generated datasets increases surgical domain knowledge compared to its base VLM by a large margin. Overall, this work validates a data-efficient and privacy-conforming way to train a surgical domain optimized, locally deployable VLM for surgical scene understanding.

Figures (2)

• Figure 1: Overview of our proposed training framework for training a surgical domain optimized, locally deployable Vision Language Model (VLM).
• Figure 2: Qualitative results of all utilized VLMs in this work. Each model is given the input image with the highlighted structure as well as the Prompt.