SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding

TL;DR

The paper addresses the limitation of frame-based surgical VQA by introducing SurgViVQA, a temporally aware VideoQA system that fuses short video clips with natural language questions via a Masked Video–Text Encoder and decodes answers with a LoRA-tuned LLM. It also provides REAL-Colon-VQA, a colonoscopy video dataset with temporally coherent QA, motion-related and diagnostic annotations, and out-of-template evaluation. Empirical results on REAL-Colon-VQA and EndoVis18-VQA show SurgViVQA significantly outperforms image-based baselines in keyword grounding and semantic metrics, with improvements around $+11\%$ and $+9\%$ in keyword accuracy, respectively. Ablation studies highlight the value of tube-masked video encoders and a keyword-penalized loss, advancing temporally-aware understanding in surgical VideoQA and enabling more reliable intraoperative decision support.

Abstract

Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation. We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video--Text Encoder to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions, which a fine-tuned large language model (LLM) then decodes into coherent answers. To evaluate its performance, we curated REAL-Colon-VQA, a colonoscopic video dataset that includes motion-related questions and diagnostic attributes, as well as out-of-template questions with rephrased or semantically altered formulations to assess model robustness. Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models, particularly in keyword accuracy, improving over PitVQA by +11\% on REAL-Colon-VQA and +9\% on EndoVis18-VQA. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing. SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally-aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts more effectively. Code and dataset available at https://github.com/madratak/SurgViVQA.

Figures (3)

• Figure 1: SurgViVQA: The architecture integrates a Masked Video–Text Encoder with a LoRA-adapted LLM to generate context-grounded answers from surgical videos.
• Figure 2: Example from the REAL-Colon-VQA dataset.
• Figure 3: qualitative comparison of predictions from all models on in-template and out-of-template example from the REAL-Colon-VQA dataset