Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest

Xupeng Chen, Zhixin Lai, Kangrui Ruan, Shichu Chen, Jiaxiang Liu, Zuozhu Liu

TL;DR

R-LLaVA addresses a key gap in medical visual question answering by injecting doctor-identified regions of interest into the image modality via CLIP and training a LLaVA-based model to attend to these ROIs. The authors implement a two-stage training pipeline—Stage I pretraining on a reduced medical caption corpus and Stage II instruction tuning with Visual RoI prompts—alongside a dataset reconstruction that introduces RoI VQA tasks. Across four Med-VQA benchmarks and a newly introduced ROI-focused multiple-choice dataset, R-LLaVA achieves state-of-the-art results, with strong performance on SLAKE-EN and VQA-Med 2019, corroborated by qualitative analyses. Ablation studies validate the necessity of both training stages and the Visual RoI approach, demonstrating that ROI-aware prompts and dynamic ROI blending consistently boost performance while highlighting practical considerations such as annotation needs and computational cost.

Abstract

Artificial intelligence has made significant strides in medical visual question answering (Med-VQA), yet prevalent studies often interpret images holistically, overlooking the visual regions of interest that may contain crucial information, potentially aligning with a doctor's prior knowledge that can be incorporated with minimal annotations (e.g., bounding boxes). To address this gap, this paper introduces R-LLaVA, designed to enhance biomedical VQA understanding by integrating simple medical annotations as prior knowledge directly into the image space through CLIP. These annotated visual regions of interest are then fed into the LLaVA model during training, aiming to enrich the model's understanding of biomedical queries. Experimental evaluation on four standard Med-VQA datasets demonstrates R-LLaVA's superiority over existing state-of-the-art (SoTA) methods. Additionally, to verify the model's capability in visual comprehension, a novel multiple-choice medical visual understanding dataset is introduced, confirming the positive impact of focusing on visual regions of interest in advancing biomedical VQA understanding.

R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest

TL;DR

R-LLaVA addresses a key gap in medical visual question answering by injecting doctor-identified regions of interest into the image modality via CLIP and training a LLaVA-based model to attend to these ROIs. The authors implement a two-stage training pipeline—Stage I pretraining on a reduced medical caption corpus and Stage II instruction tuning with Visual RoI prompts—alongside a dataset reconstruction that introduces RoI VQA tasks. Across four Med-VQA benchmarks and a newly introduced ROI-focused multiple-choice dataset, R-LLaVA achieves state-of-the-art results, with strong performance on SLAKE-EN and VQA-Med 2019, corroborated by qualitative analyses. Ablation studies validate the necessity of both training stages and the Visual RoI approach, demonstrating that ROI-aware prompts and dynamic ROI blending consistently boost performance while highlighting practical considerations such as annotation needs and computational cost.

Abstract

Artificial intelligence has made significant strides in medical visual question answering (Med-VQA), yet prevalent studies often interpret images holistically, overlooking the visual regions of interest that may contain crucial information, potentially aligning with a doctor's prior knowledge that can be incorporated with minimal annotations (e.g., bounding boxes). To address this gap, this paper introduces R-LLaVA, designed to enhance biomedical VQA understanding by integrating simple medical annotations as prior knowledge directly into the image space through CLIP. These annotated visual regions of interest are then fed into the LLaVA model during training, aiming to enrich the model's understanding of biomedical queries. Experimental evaluation on four standard Med-VQA datasets demonstrates R-LLaVA's superiority over existing state-of-the-art (SoTA) methods. Additionally, to verify the model's capability in visual comprehension, a novel multiple-choice medical visual understanding dataset is introduced, confirming the positive impact of focusing on visual regions of interest in advancing biomedical VQA understanding.

Paper Structure

This paper contains 21 sections, 2 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Medical Dataset Reconstruction with Region-of-Interest (RoI) VQA
  4. R-LLaVA Training Stage I: Pre-training
  5. R-LLaVA Training Stage II: Instruction Tuning with Visual RoI Approach
  6. Experiment
  7. Dataset
  8. Metrics
  9. Training Strategy and Setting
  10. Main Results
  11. Quantitative Comparison
  12. Qualitative Comparison
  13. Ablation Study
  14. Model Selection
  15. Two-stage Training Strategy
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Integration of doctor-guided Region-of-Interest (RoI) and original image for multimodal conversational assistance. Specifically, a Doctor-Guided ROI, which is highlighted in the red box, is overlaid onto the original image in line with the clinician's diagnostic approach.
  • Figure 2: Region-of-Interest (RoI) QA from Reconstructed VQA Dataset
  • Figure 3: R-LLaVA Training Pipeline (Stage I and Stage II)
  • Figure 4: Qualitative comparison of medical visual question answering on three types of questions from SLAKE dataset.