Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Targeted Visual Prompting for Medical Visual Question Answering

Sergio Tascon-Morales, Pablo Márquez-Neila, Raphael Sznitman

TL;DR

Medical-VQA faces data scarcity and diverse imaging modalities, hindering reliable visual grounding in existing systems. The authors propose Targeted Visual Prompting for multimodal LLMs, which encodes both a region-in-context and an isolated-region prompt to enable region-focused questions without adding model parameters. Evaluations on DME-VQA, RIS-VQA, and INSEGCAT-VQA show consistent accuracy and F1 gains over strong baselines, highlighting improved localization and robustness to context versus region cues. This approach enhances explainability and practical utility for clinical assistants and medical education by enabling modular, region-specific analysis of medical images.

Abstract

With growing interest in recent years, medical visual question answering (Med-VQA) has rapidly evolved, with multimodal large language models (MLLMs) emerging as an alternative to classical model architectures. Specifically, their ability to add visual information to the input of pre-trained LLMs brings new capabilities for image interpretation. However, simple visual errors cast doubt on the actual visual understanding abilities of these models. To address this, region-based questions have been proposed as a means to assess and enhance actual visual understanding through compositional evaluation. To combine these two perspectives, this paper introduces targeted visual prompting to equip MLLMs with region-based questioning capabilities. By presenting the model with both the isolated region and the region in its context in a customized visual prompt, we show the effectiveness of our method across multiple datasets while comparing it to several baseline models. Our code and data are available at https://github.com/sergiotasconmorales/locvqallm.

Targeted Visual Prompting for Medical Visual Question Answering

TL;DR

Medical-VQA faces data scarcity and diverse imaging modalities, hindering reliable visual grounding in existing systems. The authors propose Targeted Visual Prompting for multimodal LLMs, which encodes both a region-in-context and an isolated-region prompt to enable region-focused questions without adding model parameters. Evaluations on DME-VQA, RIS-VQA, and INSEGCAT-VQA show consistent accuracy and F1 gains over strong baselines, highlighting improved localization and robustness to context versus region cues. This approach enhances explainability and practical utility for clinical assistants and medical education by enabling modular, region-specific analysis of medical images.

Abstract

With growing interest in recent years, medical visual question answering (Med-VQA) has rapidly evolved, with multimodal large language models (MLLMs) emerging as an alternative to classical model architectures. Specifically, their ability to add visual information to the input of pre-trained LLMs brings new capabilities for image interpretation. However, simple visual errors cast doubt on the actual visual understanding abilities of these models. To address this, region-based questions have been proposed as a means to assess and enhance actual visual understanding through compositional evaluation. To combine these two perspectives, this paper introduces targeted visual prompting to equip MLLMs with region-based questioning capabilities. By presenting the model with both the isolated region and the region in its context in a customized visual prompt, we show the effectiveness of our method across multiple datasets while comparing it to several baseline models. Our code and data are available at https://github.com/sergiotasconmorales/locvqallm.
Paper Structure (8 sections, 4 equations, 5 figures, 2 tables)

This paper contains 8 sections, 4 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments and results
  4. Implementation details:
  5. Results
  6. Conclusions
  7. Acknowledgments.
  8. Disclosure of Interests.

Figures (5)

  • Figure 1: Examples of visual understanding failures using GPT-4V for the VQA task (Examples taken from tong2024eyes).
  • Figure 2: Our customized targeted visual prompt is created by providing the model with the region in context, as well as an isolated version of the region. Visual tokens are projected to the input space of the LLM and concatenated with the instruction tokens.
  • Figure 3: Example input images and questions for evaluated baselines. In the baseline "Region in text," the digits are separated to provide a fair scenario to the LLM.
  • Figure 4: Qualitative examples on the DME-VQA (first row), RIS-VQA (second row), and INSEGCAT-VQA (third row) datasets.
  • Figure 5: Error analysis by region location for the four strongest baselines. The maps are obtained by adding binary masks representing the regions for all QA pairs in each category and then normalizing. Top: DME-VQA dataset. Bottom: INSEGCAT-VQA dataset.