Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering

Mai A. Shaaban, Tausifa Jan Saleem, Vijay Ram Papineni, Mohammad Yaqub

TL;DR

This paper tackles factual inaccuracies in Medical Visual Question Answering (MedVQA) by introducing MOTOR, a multimodal retrieval-augmented framework that grounds and re-ranks external context using grounded captions and optimal transport. The method end-to-end generates grounded captions, retrieves multimodal candidates, and applies OT-based re-ranking to align textual and visual information before feeding the top contexts into a frozen VLM. Across MedVQA benchmarks, MOTOR yields a substantial average improvement of +6.45% over state-of-the-art methods, with ablations confirming the value of integrating both textual and visual signals. The results, supported by automated and expert radiologist evaluations, demonstrate enhanced factual accuracy and robust retrieval quality, suggesting strong potential for broader adoption in medical imaging tasks and future extensions to other modalities.

Abstract

Medical visual question answering (MedVQA) plays a vital role in clinical decision-making by providing contextually rich answers to image-based queries. Although vision-language models (VLMs) are widely used for this task, they often generate factually incorrect answers. Retrieval-augmented generation addresses this challenge by providing information from external sources, but risks retrieving irrelevant context, which can degrade the reasoning capabilities of VLMs. Re-ranking retrievals, as introduced in existing approaches, enhances retrieval relevance by focusing on query-text alignment. However, these approaches neglect the visual or multimodal context, which is particularly crucial for medical diagnosis. We propose MOTOR, a novel multimodal retrieval and re-ranking approach that leverages grounded captions and optimal transport. It captures the underlying relationships between the query and the retrieved context based on textual and visual information. Consequently, our approach identifies more clinically relevant contexts to augment the VLM input. Empirical analysis and human expert evaluation demonstrate that MOTOR achieves higher accuracy on MedVQA datasets, outperforming state-of-the-art methods by an average of 6.45%. Code is available at https://github.com/BioMedIA-MBZUAI/MOTOR.

MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering

TL;DR

This paper tackles factual inaccuracies in Medical Visual Question Answering (MedVQA) by introducing MOTOR, a multimodal retrieval-augmented framework that grounds and re-ranks external context using grounded captions and optimal transport. The method end-to-end generates grounded captions, retrieves multimodal candidates, and applies OT-based re-ranking to align textual and visual information before feeding the top contexts into a frozen VLM. Across MedVQA benchmarks, MOTOR yields a substantial average improvement of +6.45% over state-of-the-art methods, with ablations confirming the value of integrating both textual and visual signals. The results, supported by automated and expert radiologist evaluations, demonstrate enhanced factual accuracy and robust retrieval quality, suggesting strong potential for broader adoption in medical imaging tasks and future extensions to other modalities.

Abstract

Medical visual question answering (MedVQA) plays a vital role in clinical decision-making by providing contextually rich answers to image-based queries. Although vision-language models (VLMs) are widely used for this task, they often generate factually incorrect answers. Retrieval-augmented generation addresses this challenge by providing information from external sources, but risks retrieving irrelevant context, which can degrade the reasoning capabilities of VLMs. Re-ranking retrievals, as introduced in existing approaches, enhances retrieval relevance by focusing on query-text alignment. However, these approaches neglect the visual or multimodal context, which is particularly crucial for medical diagnosis. We propose MOTOR, a novel multimodal retrieval and re-ranking approach that leverages grounded captions and optimal transport. It captures the underlying relationships between the query and the retrieved context based on textual and visual information. Consequently, our approach identifies more clinically relevant contexts to augment the VLM input. Empirical analysis and human expert evaluation demonstrate that MOTOR achieves higher accuracy on MedVQA datasets, outperforming state-of-the-art methods by an average of 6.45%. Code is available at https://github.com/BioMedIA-MBZUAI/MOTOR.

Paper Structure

This paper contains 6 sections, 4 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experimental Setup
  4. Results and Discussion
  5. Conclusion
  6. Disclosure of Interests.

Figures (4)

  • Figure 1: Red: misleading context is retrieved, causing an incorrect answer. Green: relevant context is prioritized by re-ranking initial retrievals, causing a correct answer.
  • Figure 2: MOTOR: (1) end-to-end generation of grounded captions for query images and database images; (2) retrieving top-$k$ similar elements using pre-computed image embeddings; (3) finding retrieved elements $R \in \mathcal{R}$ that have minimal OT cost w.r.t. $Q$; and (4) passing $Q$ and the most relevant reports $\mathcal{M}$ to a VLM to generate an answer.
  • Figure 3: Proportion of samples with modified retrieved context after re-ranking.
  • Figure 4: Comparison of the clinical relevance of retrievals: before and after re-ranking.