PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering

Abstract

Trustworthy biomedical question answering (QA) systems must not only provide accurate answers but also justify them with current, verifiable evidence. Retrieval-augmented approaches partially address this gap but lack mechanisms to iteratively refine poor queries, whereas self-reflection methods kick in only after full retrieval is completed. In this context, we introduce PubMed Reasoner, a biomedical QA agent composed of three stages: self-critic query refinement evaluates MeSH terms for coverage, alignment, and redundancy to enhance PubMed queries based on partial (metadata) retrieval; reflective retrieval processes articles in batches until sufficient evidence is gathered; and evidence-grounded response generation produces answers with explicit citations. PubMed Reasoner with a GPT-4o backbone achieves 78.32% accuracy on PubMedQA, slightly surpassing human experts, and showing consistent gains on MMLU Clinical Knowledge. Moreover, LLM-as-judge evaluations prefer our responses across: reasoning soundness, evidence grounding, clinical relevance, and trustworthiness. By orchestrating retrieval-first reasoning over authoritative sources, our approach provides practical assistance to clinicians and biomedical researchers while controlling compute and token costs.

Figures (3)

• Figure 1: RAG, Self-reflection vs. PubMed Reasoner. (a) RAG baselines: uses few-shot exemplars or custom databases but lack retrieval feedback. (b) Self-reflection agents: inspired our proposal; generates responses first and only reflects after completion. (c) PubMed Reasoner: a search-first approach that performs self-critic query refinement, reflective article retrieval in batches w/ early stopping once evidence is sufficient, and evidence-grounded response generation with explicit citations.
• Figure 2: PubMed Reasoner stages. (1) Search with Self-Critic Query Refinement. From a user question, MeSH terms and a structured query are proposed. Self-critic evaluates each term for coverage, alignment, and redundancy, iteratively refining the query. (2) Reflective Article Retrieval with Early Stopping. PubMed Reasoner queries PubMed, filters results by title/abstract, and extracts supporting evidence in batches, checking whether accumulated evidence sufficiently covers the question; if so, retrieval is terminated early to save tokens and avoid unnecessary processing. (3) Evidence-Grounded Response Generation. Retained evidence is synthesized into final answer with explicit inline citations, ensuring factual grounding and traceability. More case studies are provided in App. \ref{['app:case']}.
• Figure 3: Effectiveness of the reflective integration stage. Distribution of retrieval depth before early stopping.