Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient and Reproducible Biomedical Question Answering using Retrieval Augmented Generation

Linus Stuhlmann, Michael Alexander Saxer, Jonathan Fürst

TL;DR

The paper tackles the challenge of scalable and reproducible biomedical QA by evaluating a Retrieval-Augmented Generation (RAG) system that combines sparse lexical retrieval (BM25) with a neural cross-encoder reranker (MedCPT) and leverages both Elasticsearch and FAISS for indexing. Through a systematic comparison of retrieval strategies and data stores on PubMed data, the study identifies a hybrid BM25+MedCPT pipeline as providing strong end-to-end performance, with retrieval depth critically shaping accuracy and latency. Key findings show that retrieving 50 documents before reranking yields high accuracy (≈0.90) and manageable latency (~1.91 s), while larger depths offer diminishing returns and greater cost. The work emphasizes reproducibility by open-sourcing code and citing PubMed IDs for traceability, offering practical guidance for deploying biomedical QA systems in research and clinical settings.

Abstract

Biomedical question-answering (QA) systems require effective retrieval and generation components to ensure accuracy, efficiency, and scalability. This study systematically examines a Retrieval-Augmented Generation (RAG) system for biomedical QA, evaluating retrieval strategies and response time trade-offs. We first assess state-of-the-art retrieval methods, including BM25, BioBERT, MedCPT, and a hybrid approach, alongside common data stores such as Elasticsearch, MongoDB, and FAISS, on a ~10% subset of PubMed (2.4M documents) to measure indexing efficiency, retrieval latency, and retriever performance in the end-to-end RAG system. Based on these insights, we deploy the final RAG system on the full 24M PubMed corpus, comparing different retrievers' impact on overall performance. Evaluations of the retrieval depth show that retrieving 50 documents with BM25 before reranking with MedCPT optimally balances accuracy (0.90), recall (0.90), and response time (1.91s). BM25 retrieval time remains stable (82ms), while MedCPT incurs the main computational cost. These results highlight previously not well-known trade-offs in retrieval depth, efficiency, and scalability for biomedical QA. With open-source code, the system is fully reproducible and extensible.

Efficient and Reproducible Biomedical Question Answering using Retrieval Augmented Generation

TL;DR

The paper tackles the challenge of scalable and reproducible biomedical QA by evaluating a Retrieval-Augmented Generation (RAG) system that combines sparse lexical retrieval (BM25) with a neural cross-encoder reranker (MedCPT) and leverages both Elasticsearch and FAISS for indexing. Through a systematic comparison of retrieval strategies and data stores on PubMed data, the study identifies a hybrid BM25+MedCPT pipeline as providing strong end-to-end performance, with retrieval depth critically shaping accuracy and latency. Key findings show that retrieving 50 documents before reranking yields high accuracy (≈0.90) and manageable latency (~1.91 s), while larger depths offer diminishing returns and greater cost. The work emphasizes reproducibility by open-sourcing code and citing PubMed IDs for traceability, offering practical guidance for deploying biomedical QA systems in research and clinical settings.

Abstract

Biomedical question-answering (QA) systems require effective retrieval and generation components to ensure accuracy, efficiency, and scalability. This study systematically examines a Retrieval-Augmented Generation (RAG) system for biomedical QA, evaluating retrieval strategies and response time trade-offs. We first assess state-of-the-art retrieval methods, including BM25, BioBERT, MedCPT, and a hybrid approach, alongside common data stores such as Elasticsearch, MongoDB, and FAISS, on a ~10% subset of PubMed (2.4M documents) to measure indexing efficiency, retrieval latency, and retriever performance in the end-to-end RAG system. Based on these insights, we deploy the final RAG system on the full 24M PubMed corpus, comparing different retrievers' impact on overall performance. Evaluations of the retrieval depth show that retrieving 50 documents with BM25 before reranking with MedCPT optimally balances accuracy (0.90), recall (0.90), and response time (1.91s). BM25 retrieval time remains stable (82ms), while MedCPT incurs the main computational cost. These results highlight previously not well-known trade-offs in retrieval depth, efficiency, and scalability for biomedical QA. With open-source code, the system is fully reproducible and extensible.
Paper Structure (11 sections, 2 figures, 4 tables)

This paper contains 11 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Biomedical Question-Answering with RAG
  3. Experimental Evaluation
  4. Experimental Setup
  5. Indexing and Query Time
  6. Document Relevancy
  7. Answer Correctness of the RAG System (End-to-End)
  8. Evaluation of the Final System
  9. Effect of Retrieval Depth on Performance
  10. Analysis of Retrieval Depth Trade-offs
  11. Conclusion and Future Directions

Figures (2)

  • Figure 1: Biomedical Question Answering with Retrieval-Augmented Generation (RAG). Offline phase: biomedical documents are processed, indexed, and stored in a vector store. Online phase: users ask questions, for which a retriever retrieves relevant documents that are appended with the question and fed to an LLM. Based on the question and context, the LLM generates an answer the PubMed IDs it used.
  • Figure 2: Prompting approach for biomedical QA.