Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine

Wei Zhu

TL;DR

This work introduces MRAG-Bench, a comprehensive benchmark for evaluating Retrieval-Augmented Generation in biomedicine, and the MRAG-Toolkit to systematically study how corpus, retriever, prompting, and LLM choice shape performance. Spanning four task types in English and Chinese across 13 datasets (14,816 samples), MRAG enables multi-faceted analysis of RAG in medical QA, LFQA, IE, and LP. Key findings show that RAG generally improves LLM reliability and usefulness, with performance strongly influenced by retrieval corpus and prompting strategies, and larger models benefitting more from RAG—though readability for long-form answers may decline. By open-sourcing the MRAG-Bench data and MRAG-Toolkit under CC BY 4.0, the work provides a valuable resource for academia and industry to advance trustworthy, up-to-date biomedical AI systems.

Abstract

While Retrieval-Augmented Generation (RAG) has been swiftly adopted in scientific and clinical QA systems, a comprehensive evaluation benchmark in the medical domain is lacking. To address this gap, we introduce the Medical Retrieval-Augmented Generation (MRAG) benchmark, covering various tasks in English and Chinese languages, and building a corpus with Wikipedia and Pubmed. Additionally, we develop the MRAG-Toolkit, facilitating systematic exploration of different RAG components. Our experiments reveal that: (a) RAG enhances LLM reliability across MRAG tasks. (b) the performance of RAG systems is influenced by retrieval approaches, model sizes, and prompting strategies. (c) While RAG improves usefulness and reasoning quality, LLM responses may become slightly less readable for long-form questions. We will release the MRAG-Bench's dataset and toolkit with CCBY-4.0 license upon acceptance, to facilitate applications from both academia and industry.

MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine

TL;DR

This work introduces MRAG-Bench, a comprehensive benchmark for evaluating Retrieval-Augmented Generation in biomedicine, and the MRAG-Toolkit to systematically study how corpus, retriever, prompting, and LLM choice shape performance. Spanning four task types in English and Chinese across 13 datasets (14,816 samples), MRAG enables multi-faceted analysis of RAG in medical QA, LFQA, IE, and LP. Key findings show that RAG generally improves LLM reliability and usefulness, with performance strongly influenced by retrieval corpus and prompting strategies, and larger models benefitting more from RAG—though readability for long-form answers may decline. By open-sourcing the MRAG-Bench data and MRAG-Toolkit under CC BY 4.0, the work provides a valuable resource for academia and industry to advance trustworthy, up-to-date biomedical AI systems.

Abstract

While Retrieval-Augmented Generation (RAG) has been swiftly adopted in scientific and clinical QA systems, a comprehensive evaluation benchmark in the medical domain is lacking. To address this gap, we introduce the Medical Retrieval-Augmented Generation (MRAG) benchmark, covering various tasks in English and Chinese languages, and building a corpus with Wikipedia and Pubmed. Additionally, we develop the MRAG-Toolkit, facilitating systematic exploration of different RAG components. Our experiments reveal that: (a) RAG enhances LLM reliability across MRAG tasks. (b) the performance of RAG systems is influenced by retrieval approaches, model sizes, and prompting strategies. (c) While RAG improves usefulness and reasoning quality, LLM responses may become slightly less readable for long-form questions. We will release the MRAG-Bench's dataset and toolkit with CCBY-4.0 license upon acceptance, to facilitate applications from both academia and industry.
Paper Structure (26 sections, 7 figures, 9 tables)

This paper contains 26 sections, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. The MRAG Benchmark
  4. Constituting tasks
  5. Evaluation metrics
  6. RAG system
  7. Results
  8. Results on the closed-form tasks
  9. Results on the LFQA tasks
  10. Ablation studies
  11. Conclusions
  12. Appendix: Related works
  13. Question answering in bio-medicine
  14. MRAG-Bench datasets
  15. Previously open-sourced datasets
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Composition of tasks in our MRAG-Bench.
  • Figure 2: Framework of the MRAG toolkit, demonstrating each of its components.
  • Figure 3: Effects of #documents retrieved.
  • Figure 4: The scaling laws of LLMs on the MRAG tasks, with or without RAG.
  • Figure 5: The screenshot of the annotation web-page for quality assurance of the link prediction tasks.
  • ...and 2 more figures