Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation

TL;DR

This work addresses the domain-specific challenges of surgical video-language pretraining by proposing PeskaVLP, a framework that combines hierarchical knowledge augmentation with procedure-aware pretraining. It leverages LLMs to enrich narration, keysteps, and abstracts, producing higher-quality language supervision, and couples this with visual self-supervision through a novel LecNCE loss. A DTW-based cross-modal temporal alignment is introduced at phase- and video-level to capture sequential surgical procedures, while a clip-level objective integrates language and vision signals. Empirically, PeskaVLP achieves strong zero-shot performance in surgical phase recognition and cross-modal retrieval, and demonstrates superior generalization across procedures and centers, establishing a robust initialization for downstream surgical scene understanding tasks.

Abstract

Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data. This study aims to bridge the gap by addressing issues regarding textual information loss in surgical lecture videos and the spatial-temporal challenges of surgical VLP. We propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) framework to tackle these issues. The knowledge augmentation uses large language models (LLM) for refining and enriching surgical concepts, thus providing comprehensive language supervision and reducing the risk of overfitting. PeskaVLP combines language supervision with visual self-supervision, constructing hard negative samples and employing a Dynamic Time Warping (DTW) based loss function to effectively comprehend the cross-modal procedural alignment. Extensive experiments on multiple public surgical scene understanding and cross-modal retrieval datasets show that our proposed method significantly improves zero-shot transferring performance and offers a generalist visual representation for further advancements in surgical scene understanding.The code is available at https://github.com/CAMMA-public/SurgVLP

Figures (10)

• Figure 1: Illustration of video-language pretraining with hierarchical video-text pairs. At phase- and video-level, one parent-level text is paired to multiple child-level texts.
• Figure 2: Hierarchical Knowledge augmentation for hierarchical texts. (a) the process of building a surgical step knowledge base. (b) the process of improving hierarchical textual quality based on LLM.
• Figure 3: The pretraining pipeline of different hierarchies. We combine language supervision and visual self-supervision at clip-level pretraining. We conduct the procedure-aware contrastive learning at phase/video-level pretraining.
• Figure 4: Modality gap visualization in different hierarchical levels. It shows that our model closes the modality gap incurred from the initialization after the hierarchical pretraining.
• Figure 5: Qualitative surgical phase recognition results on hysterectomy. The y-axis is the class names. The x-axis is the probability of each class. The bottom right image shows that the pretrained model understands the blood fluid.