Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proactive Reasoning-with-Retrieval Framework for Medical Multimodal Large Language Models

Lehan Wang, Yi Qin, Honglong Yang, Xiaomeng Li

TL;DR

This work tackles the unreliability of medical multimodal LLMs by grounding reasoning in external knowledge. It introduces Med-RwR, a multimodal reasoning-with-retrieval framework trained with a two-stage reinforcement learning strategy and reinforced by a novel reward design that aligns visual and textual information during retrieval. A Confidence-Driven Image Re-retrieval (CDIR) module is added to scale test-time retrieval when confidence is low. Empirical results on multiple benchmarks show Med-RwR achieves state-of-the-art performance and good generalization to echocardiography through domain-specific knowledge bases, with publicly released data and code. Overall, the approach demonstrates that proactive, multimodal knowledge retrieval substantially enhances medical reasoning and decision support in unfamiliar domains.

Abstract

Incentivizing the reasoning ability of Multimodal Large Language Models (MLLMs) is essential for medical applications to transparently analyze medical scans and provide reliable diagnosis. However, existing medical MLLMs rely solely on internal knowledge during reasoning, leading to hallucinated reasoning and factual inaccuracies when encountering cases beyond their training scope. Although recent Agentic Retrieval-Augmented Generation (RAG) methods elicit the medical model's proactive retrieval ability during reasoning, they are confined to unimodal LLMs, neglecting the crucial visual information during reasoning and retrieval. Consequently, we propose the first Multimodal Medical Reasoning-with-Retrieval framework, Med-RwR, which actively retrieves external knowledge by querying observed symptoms or domain-specific medical concepts during reasoning. Specifically, we design a two-stage reinforcement learning strategy with tailored rewards that stimulate the model to leverage both visual diagnostic findings and textual clinical information for effective retrieval. Building on this foundation, we further propose a Confidence-Driven Image Re-retrieval (CDIR) method for test-time scaling when low prediction confidence is detected. Evaluation on various public medical benchmarks demonstrates Med-RwR's significant improvements over baseline models, proving the effectiveness of enhancing reasoning capabilities with external knowledge integration. Furthermore, Med-RwR demonstrates remarkable generalizability to unfamiliar domains, evidenced by 8.8% performance gain on our proposed EchoCardiography Benchmark (ECBench), despite the scarcity of echocardiography data in the training corpus. Our data, model, and codes will be made publicly available at https://github.com/xmed-lab/Med-RwR.

Proactive Reasoning-with-Retrieval Framework for Medical Multimodal Large Language Models

TL;DR

This work tackles the unreliability of medical multimodal LLMs by grounding reasoning in external knowledge. It introduces Med-RwR, a multimodal reasoning-with-retrieval framework trained with a two-stage reinforcement learning strategy and reinforced by a novel reward design that aligns visual and textual information during retrieval. A Confidence-Driven Image Re-retrieval (CDIR) module is added to scale test-time retrieval when confidence is low. Empirical results on multiple benchmarks show Med-RwR achieves state-of-the-art performance and good generalization to echocardiography through domain-specific knowledge bases, with publicly released data and code. Overall, the approach demonstrates that proactive, multimodal knowledge retrieval substantially enhances medical reasoning and decision support in unfamiliar domains.

Abstract

Incentivizing the reasoning ability of Multimodal Large Language Models (MLLMs) is essential for medical applications to transparently analyze medical scans and provide reliable diagnosis. However, existing medical MLLMs rely solely on internal knowledge during reasoning, leading to hallucinated reasoning and factual inaccuracies when encountering cases beyond their training scope. Although recent Agentic Retrieval-Augmented Generation (RAG) methods elicit the medical model's proactive retrieval ability during reasoning, they are confined to unimodal LLMs, neglecting the crucial visual information during reasoning and retrieval. Consequently, we propose the first Multimodal Medical Reasoning-with-Retrieval framework, Med-RwR, which actively retrieves external knowledge by querying observed symptoms or domain-specific medical concepts during reasoning. Specifically, we design a two-stage reinforcement learning strategy with tailored rewards that stimulate the model to leverage both visual diagnostic findings and textual clinical information for effective retrieval. Building on this foundation, we further propose a Confidence-Driven Image Re-retrieval (CDIR) method for test-time scaling when low prediction confidence is detected. Evaluation on various public medical benchmarks demonstrates Med-RwR's significant improvements over baseline models, proving the effectiveness of enhancing reasoning capabilities with external knowledge integration. Furthermore, Med-RwR demonstrates remarkable generalizability to unfamiliar domains, evidenced by 8.8% performance gain on our proposed EchoCardiography Benchmark (ECBench), despite the scarcity of echocardiography data in the training corpus. Our data, model, and codes will be made publicly available at https://github.com/xmed-lab/Med-RwR.
Paper Structure (47 sections, 7 equations, 17 figures, 14 tables, 1 algorithm)

This paper contains 47 sections, 7 equations, 17 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Medical Multimodal Large Language Models
  4. Agentic Search and Retrieval
  5. Methodology
  6. Reasoning-with-Retrieval Environment
  7. Multimodal Medical Reasoning-with-Retrieval Framework
  8. Problem Formulation
  9. Reward Design
  10. Model Training
  11. Confidence-Driven Test-Time Computation Scaling via Image Re-retrieval
  12. Experiments
  13. Experimental Settings
  14. Main Results
  15. Ablation Study
  16. ...and 32 more sections

Figures (17)

  • Figure 1: (a) Comparison between the generated outputs of a recent reasoning model, Chiron-o1 sun2025enhancing, and the rollout process of our proposed Reasoning-with-Retrieval framework, which includes a trajectory of think, query and retrieve steps. Texts in green denote the correct statements, red indicates erroneous outputs, and brown annotates the authoritative knowledge. (b) Comparison between the effectiveness of retrieval between text-only agentic RAG system for diagnosis, Deep-DxSearch zheng2025end, and our proposed multimodal Med-RwR.
  • Figure 2: Overall Illustration of the Reasoning-with-Retrieval framework. During training, the model learns to generate reasoning steps while actively retrieving from a medical knowledge base. During inference, the Confidence-Driven Image Re-retrieval scheme would be triggered to augment less informative retrieval when low confidence is detected.
  • Figure 3: (a) Ablation Study of the Reasoning-with-Retrieval Capability: comparing the vanilla model, the reasoning model, and the reasoning-with-retrieval model on two architectures, QwenVL-2.5-7B (left) and Lingshu-7B (right). (b) Comparison with Training-Free RAG methods, which apply the original question, MLLM-reformulated question and MLLM-generated caption as the query, respectively. (c) Confidence Gain after Confidence-Driven Image Re-retrieval.
  • Figure 4: Modality and Body Structure Distribution of Our Constructed Multimodal Dataset.
  • Figure 5: Illustration of the Data Synthesis Process.
  • ...and 12 more figures