BioMedSearch: A Multi-Source Biomedical Retrieval Framework Based on LLMs

TL;DR

BioMedSearch tackles biomedical LLM hallucinations by integrating literature databases, protein knowledge bases, and real-time web search into a multi-source retrieval framework. It introduces a biomedical search planner that decomposes queries into sub-queries and constructs a DAG to guide retrieval across literature, protein databases, and the web, followed by a summary report generator that synthesizes structured answers. Evaluated on the BioMedMCQs benchmark (3 levels, 3,000 questions), BioMedSearch consistently surpasses baselines in accuracy across all levels, demonstrating improved semantic alignment and reasoning. The approach enhances practical biomedical knowledge discovery by enabling real-time access and domain-specific filtering, with implications for more reliable biomedical information retrieval and decision support.

Abstract

Biomedical queries often rely on a deep understanding of specialized knowledge such as gene regulatory mechanisms and pathological processes of diseases. They require detailed analysis of complex physiological processes and effective integration of information from multiple data sources to support accurate retrieval and reasoning. Although large language models (LLMs) perform well in general reasoning tasks, their generated biomedical content often lacks scientific rigor due to the inability to access authoritative biomedical databases and frequently fabricates protein functions, interactions, and structural details that deviate from authentic information. Therefore, we present BioMedSearch, a multi-source biomedical information retrieval framework based on LLMs. The method integrates literature retrieval, protein database and web search access to support accurate and efficient handling of complex biomedical queries. Through sub-queries decomposition, keywords extraction, task graph construction, and multi-source information filtering, BioMedSearch generates high-quality question-answering results. To evaluate the accuracy of question answering, we constructed a multi-level dataset, BioMedMCQs, consisting of 3,000 questions. The dataset covers three levels of reasoning: mechanistic identification, non-adjacent semantic integration, and temporal causal reasoning, and is used to assess the performance of BioMedSearch and other methods on complex QA tasks. Experimental results demonstrate that BioMedSearch consistently improves accuracy over all baseline models across all levels. Specifically, at Level 1, the average accuracy increases from 59.1% to 91.9%; at Level 2, it rises from 47.0% to 81.0%; and at the most challenging Level 3, the average accuracy improves from 36.3% to 73.4%. The code and BioMedMCQs are available at: https://github.com/CyL-ucas/BioMed_Search

Figures (4)

• Figure 1: Demonstration of the BioMedSearch workflow using an example query. The purple boxes represent the simulated user input and its corresponding flowchart, including the decomposition of the user query into sub-queries, further keywords extraction, and the planning of appropriate retrieval paths based on these keywords. Subsequently, information retrieval and filtering are performed to obtain relevant results, which are then used to select suitable sub-answers and generate a structured report for user feedback. The blue boxes indicate the core prompting strategies designed for the LLM at each stage of the workflow.
• Figure 2: Automated BioMedMCQs generation pipeline based on LLM and literature summarization. This figure illustrates a two-stage pipeline for automatically generating BioMedMCQs using LLM. In the first stage, the LLM generates a random biomedical topic, extracts relevant keywords, retrieves and filters top k literature, and produces a summary report. In the second stage, the report content is used to construct Level 1–3 MCQs, which are iteratively verified and regenerated until the required number of questions is met.
• Figure 3: Illustration of the answer generation on BioMedMCQs using LLMs.
• Figure 4: Comparison of average accuracy across different methods on BioMedMCQs at three reasoning levels (Level 1–3). For each method, the reported value represents the mean accuracy across all LLMs at the corresponding level, reflecting overall performance in multi-level biomedical MCQs.