Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA

A H M Rezaul Karim, Ozlem Uzuner

TL;DR

This work demonstrates that a general-domain instruction-tuned LLM combined with a lightweight retrieval-augmented generation layer can tackle multimodal wound-care MedVQA without domain-specific fine-tuning. By retrieving a couple of in-domain textual and visual exemplars and fusing them into prompts, the approach grounding improves reasoning, schema adherence, and output quality. In extensive experiments on MEDIQA-WV 2025, the MasonNLP system achieved top-five leaderboard performance (Avg 41.37%), with ablation showing clear gains from both exemplar-based prompting and multimodal retrieval. The findings suggest that simple, reproducible RAG pipelines can serve as effective baselines for multimodal clinical NLP tasks, balancing generation flexibility with grounding and interpretability.

Abstract

Medical Visual Question Answering (MedVQA) enables natural language queries over medical images to support clinical decision-making and patient care. The MEDIQA-WV 2025 shared task addressed wound-care VQA, requiring systems to generate free-text responses and structured wound attributes from images and patient queries. We present the MasonNLP system, which employs a general-domain, instruction-tuned large language model with a retrieval-augmented generation (RAG) framework that incorporates textual and visual examples from in-domain data. This approach grounds outputs in clinically relevant exemplars, improving reasoning, schema adherence, and response quality across dBLEU, ROUGE, BERTScore, and LLM-based metrics. Our best-performing system ranked 3rd among 19 teams and 51 submissions with an average score of 41.37%, demonstrating that lightweight RAG with general-purpose LLMs -- a minimal inference-time layer that adds a few relevant exemplars via simple indexing and fusion, with no extra training or complex re-ranking -- provides a simple and effective baseline for multimodal clinical NLP tasks.

Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA

TL;DR

This work demonstrates that a general-domain instruction-tuned LLM combined with a lightweight retrieval-augmented generation layer can tackle multimodal wound-care MedVQA without domain-specific fine-tuning. By retrieving a couple of in-domain textual and visual exemplars and fusing them into prompts, the approach grounding improves reasoning, schema adherence, and output quality. In extensive experiments on MEDIQA-WV 2025, the MasonNLP system achieved top-five leaderboard performance (Avg 41.37%), with ablation showing clear gains from both exemplar-based prompting and multimodal retrieval. The findings suggest that simple, reproducible RAG pipelines can serve as effective baselines for multimodal clinical NLP tasks, balancing generation flexibility with grounding and interpretability.

Abstract

Medical Visual Question Answering (MedVQA) enables natural language queries over medical images to support clinical decision-making and patient care. The MEDIQA-WV 2025 shared task addressed wound-care VQA, requiring systems to generate free-text responses and structured wound attributes from images and patient queries. We present the MasonNLP system, which employs a general-domain, instruction-tuned large language model with a retrieval-augmented generation (RAG) framework that incorporates textual and visual examples from in-domain data. This approach grounds outputs in clinically relevant exemplars, improving reasoning, schema adherence, and response quality across dBLEU, ROUGE, BERTScore, and LLM-based metrics. Our best-performing system ranked 3rd among 19 teams and 51 submissions with an average score of 41.37%, demonstrating that lightweight RAG with general-purpose LLMs -- a minimal inference-time layer that adds a few relevant exemplars via simple indexing and fusion, with no extra training or complex re-ranking -- provides a simple and effective baseline for multimodal clinical NLP tasks.
Paper Structure (23 sections, 2 equations, 2 figures, 4 tables)

This paper contains 23 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Description
  4. Dataset
  5. Dataset Analysis
  6. Methodology
  7. Model Configuration
  8. Prompt Design
  9. Zero-shot prompting.
  10. Few-shot prompting.
  11. Retrieval-augmented prompting.
  12. Experimental Setup
  13. Post-processing
  14. Evaluation
  15. Results and Discussion
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Task overview for MEDIQA-WV 2025. Inputs: wound images and a patient query. Outputs: free-text answer with structured wound attributes
  • Figure 2: Structured prompt with retrieved exemplars and the expected output schema.