Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery

Runlong He, Mengya Xu, Adrito Das, Danyal Z. Khan, Sophia Bano, Hani J. Marcus, Danail Stoyanov, Matthew J. Clarkson, Mobarakol Islam

TL;DR

PitVQA tackles the scarcity of large, diverse datasets for surgical VQA and the challenging fusion of image and text in endonasal pituitary surgery. It introduces the PitVQA dataset and PitVQA-Net, an image-grounded text embedding built on a GPT2 backbone with an excitation block, achieving improved balanced accuracy over baselines on both PitVQA and EndoVis18-VQA. The approach leverages cross-attention between image features and questions and uses a gated excitation head to enhance robust VQA in complex surgical contexts. The work demonstrates potential for real-time intraoperative decision support and intuitive surgeon–AI interaction, with future directions including dataset expansion and real-time deployment testing in the OR.

Abstract

Visual Question Answering (VQA) within the surgical domain, utilizing Large Language Models (LLMs), offers a distinct opportunity to improve intra-operative decision-making and facilitate intuitive surgeon-AI interaction. However, the development of LLMs for surgical VQA is hindered by the scarcity of diverse and extensive datasets with complex reasoning tasks. Moreover, contextual fusion of the image and text modalities remains an open research challenge due to the inherent differences between these two types of information and the complexity involved in aligning them. This paper introduces PitVQA, a novel dataset specifically designed for VQA in endonasal pituitary surgery and PitVQA-Net, an adaptation of the GPT2 with a novel image-grounded text embedding for surgical VQA. PitVQA comprises 25 procedural videos and a rich collection of question-answer pairs spanning crucial surgical aspects such as phase and step recognition, context understanding, tool detection and localization, and tool-tissue interactions. PitVQA-Net consists of a novel image-grounded text embedding that projects image and text features into a shared embedding space and GPT2 Backbone with an excitation block classification head to generate contextually relevant answers within the complex domain of endonasal pituitary surgery. Our image-grounded text embedding leverages joint embedding, cross-attention and contextual representation to understand the contextual relationship between questions and surgical images. We demonstrate the effectiveness of PitVQA-Net on both the PitVQA and the publicly available EndoVis18-VQA dataset, achieving improvements in balanced accuracy of 8% and 9% over the most recent baselines, respectively. Our code and dataset is available at https://github.com/mobarakol/PitVQA.

PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery

TL;DR

PitVQA tackles the scarcity of large, diverse datasets for surgical VQA and the challenging fusion of image and text in endonasal pituitary surgery. It introduces the PitVQA dataset and PitVQA-Net, an image-grounded text embedding built on a GPT2 backbone with an excitation block, achieving improved balanced accuracy over baselines on both PitVQA and EndoVis18-VQA. The approach leverages cross-attention between image features and questions and uses a gated excitation head to enhance robust VQA in complex surgical contexts. The work demonstrates potential for real-time intraoperative decision support and intuitive surgeon–AI interaction, with future directions including dataset expansion and real-time deployment testing in the OR.

Abstract

Visual Question Answering (VQA) within the surgical domain, utilizing Large Language Models (LLMs), offers a distinct opportunity to improve intra-operative decision-making and facilitate intuitive surgeon-AI interaction. However, the development of LLMs for surgical VQA is hindered by the scarcity of diverse and extensive datasets with complex reasoning tasks. Moreover, contextual fusion of the image and text modalities remains an open research challenge due to the inherent differences between these two types of information and the complexity involved in aligning them. This paper introduces PitVQA, a novel dataset specifically designed for VQA in endonasal pituitary surgery and PitVQA-Net, an adaptation of the GPT2 with a novel image-grounded text embedding for surgical VQA. PitVQA comprises 25 procedural videos and a rich collection of question-answer pairs spanning crucial surgical aspects such as phase and step recognition, context understanding, tool detection and localization, and tool-tissue interactions. PitVQA-Net consists of a novel image-grounded text embedding that projects image and text features into a shared embedding space and GPT2 Backbone with an excitation block classification head to generate contextually relevant answers within the complex domain of endonasal pituitary surgery. Our image-grounded text embedding leverages joint embedding, cross-attention and contextual representation to understand the contextual relationship between questions and surgical images. We demonstrate the effectiveness of PitVQA-Net on both the PitVQA and the publicly available EndoVis18-VQA dataset, achieving improvements in balanced accuracy of 8% and 9% over the most recent baselines, respectively. Our code and dataset is available at https://github.com/mobarakol/PitVQA.
Paper Structure (13 sections, 3 figures, 3 tables)

This paper contains 13 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries
  4. Proposed Method: PitVQA
  5. PitVQA Dataset:
  6. PitVQA-Net:
  7. Experiments and Results
  8. Dataset
  9. Implementation Details
  10. Evaluation Metrics
  11. Results
  12. Ablation Study
  13. Discussion and Conclusion

Figures (3)

  • Figure 1: PitVQA dataset of visual questions answering for pituitary surgery. There are overall 59 classes in the 6 class categories of phases, steps, instruments, quantity, positions and operation notes.
  • Figure 2: PitVQA-Net: The network forms of Image-grounded Text Embedding, GPT2 Backbone and Classification Head. The image-grounded text embedding leverages joint embedding, cross-attention and contextual representation.
  • Figure 3: Qualitative visualization of our model prediction comparing with closely related works with datasets of our PitVQA and EndoVis18-VQA.