Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Med-R$^2$: Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine

Keer Lu, Zheng Liang, Da Pan, Shusen Zhang, Guosheng Dong, Zhonghai Wu, Huang Leng, Bin Cui, Wentao Zhang

TL;DR

Med-R^2 proposes an Evidence-Based Medicine–driven architecture that tightly couples retrieval, appraisal, and reasoning to create trustworthy LLM physicians. By reformulating clinical queries, using a dual dense-sparse knowledge base, and applying a coarse-to-fine evidence reranking plus chain-of-thought reasoning, it delivers strong performance improvements over vanilla RAG and even some fine-tuning baselines without additional training costs. Experimental results across multiple medical benchmarks show that Med-R^2, especially when paired with larger models (e.g., LLaMA3.1-70B), can surpass frontier models on average and demonstrates particularly large gains for smaller models. The framework emphasizes interpretability and robustness through explicit evidence hierarchy, usefulness scoring, and iterative evidence assessment, suggesting practical impact for medical AI systems requiring reliable evidence-based decisions.

Abstract

Large Language Models (LLMs) have exhibited remarkable capabilities in clinical scenarios. Despite their potential, existing works face challenges when applying LLMs to medical settings. Strategies relying on training with medical datasets are highly cost-intensive and may suffer from outdated training data. Leveraging external knowledge bases is a suitable alternative, yet it faces obstacles such as limited retrieval precision and poor effectiveness in answer extraction. These issues collectively prevent LLMs from demonstrating the expected level of proficiency in mastering medical expertise. To address these challenges, we introduce Med-R^2, a novel LLM physician framework that adheres to the Evidence-Based Medicine (EBM) process, efficiently integrating retrieval mechanisms as well as the selection and reasoning processes of evidence, thereby enhancing the problem-solving capabilities of LLMs in healthcare scenarios and fostering a trustworthy LLM physician. Our comprehensive experiments indicate that Med-R^2 achieves a 13.27\% improvement over vanilla RAG methods and even a 4.55\% enhancement compared to fine-tuning strategies, without incurring additional training costs. Furthermore, we find that our LLaMA3.1-70B + Med-R$^2$ surpasses frontier models, including GPT-4o, Claude3.5-Sonnet and DeepSeek-V3 by 1.05\%, 6.14\% and 1.91\%. Med-R$^2$ effectively enhances the capabilities of LLMs in the medical domain.

Med-R$^2$: Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine

TL;DR

Med-R^2 proposes an Evidence-Based Medicine–driven architecture that tightly couples retrieval, appraisal, and reasoning to create trustworthy LLM physicians. By reformulating clinical queries, using a dual dense-sparse knowledge base, and applying a coarse-to-fine evidence reranking plus chain-of-thought reasoning, it delivers strong performance improvements over vanilla RAG and even some fine-tuning baselines without additional training costs. Experimental results across multiple medical benchmarks show that Med-R^2, especially when paired with larger models (e.g., LLaMA3.1-70B), can surpass frontier models on average and demonstrates particularly large gains for smaller models. The framework emphasizes interpretability and robustness through explicit evidence hierarchy, usefulness scoring, and iterative evidence assessment, suggesting practical impact for medical AI systems requiring reliable evidence-based decisions.

Abstract

Large Language Models (LLMs) have exhibited remarkable capabilities in clinical scenarios. Despite their potential, existing works face challenges when applying LLMs to medical settings. Strategies relying on training with medical datasets are highly cost-intensive and may suffer from outdated training data. Leveraging external knowledge bases is a suitable alternative, yet it faces obstacles such as limited retrieval precision and poor effectiveness in answer extraction. These issues collectively prevent LLMs from demonstrating the expected level of proficiency in mastering medical expertise. To address these challenges, we introduce Med-R^2, a novel LLM physician framework that adheres to the Evidence-Based Medicine (EBM) process, efficiently integrating retrieval mechanisms as well as the selection and reasoning processes of evidence, thereby enhancing the problem-solving capabilities of LLMs in healthcare scenarios and fostering a trustworthy LLM physician. Our comprehensive experiments indicate that Med-R^2 achieves a 13.27\% improvement over vanilla RAG methods and even a 4.55\% enhancement compared to fine-tuning strategies, without incurring additional training costs. Furthermore, we find that our LLaMA3.1-70B + Med-R surpasses frontier models, including GPT-4o, Claude3.5-Sonnet and DeepSeek-V3 by 1.05\%, 6.14\% and 1.91\%. Med-R effectively enhances the capabilities of LLMs in the medical domain.
Paper Structure (32 sections, 4 equations, 11 figures, 8 tables, 2 algorithms)

This paper contains 32 sections, 4 equations, 11 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work and Discussions
  3. Med-R$^2$
  4. Question Formulation
  5. Evidence Searching && Appraising
  6. Evidence Retriever
  7. Evidence Reranker
  8. Evidence Applying
  9. Effect Assessment
  10. Experiments and Results
  11. Experimental Setup
  12. Baselines
  13. Main Results
  14. Ablations and Analysis Across Scales
  15. Discussion
  16. ...and 17 more sections

Figures (11)

  • Figure 1: Comparison of Med-R$^2$ with existing strategies for medical problem-solving.
  • Figure 2: An illustration of Med-R$^2$'s process, adhering to the Evidence-Based Medicine (EBM) workflow. We first categorize the query by EBM and general question types. Queries are then reformulated according to established EBM classification templates to ensure precision and relevance. In the evidence searching and appraising stages, we employ a coarse-to-fine strategy to retrieve, filter, and re-rank the evidence documents within the knowledge base. CoT sequences are then generated from processed evidence to refine retrieval space, iterating to ensure the robustness.
  • Figure 3: Query category of MedMCQA. We employ a logarithmic scale (base 10) on the z-axis, ranging from 1 to 40000, to represent the wide range of values.
  • Figure 4: Hierarchy of evidence. The base of the pyramid represents the lowest quality, while the apex denotes the highest.
  • Figure 5: Query Document Projection.
  • ...and 6 more figures