Talk Before You Retrieve: Agent-Led Discussions for Better RAG in Medical QA

TL;DR

Medical QA with LLMs suffers from hallucinations and outdated knowledge; Retrieval-Augmented Generation (RAG) helps but lacks human-like retrieval and post-retrieval verification. Discuss-RAG introduces a two-part system: a multi-turn, agent-based discussion and summarization to produce context-rich background before retrieval, and a post-retrieval verifier that filters retrieved snippets using the distilled summary, with a fallback mechanism. Across four medical QA benchmarks (MMLU-Med, MedQA-US, BioASQ, PubMedQA) using a fixed corpus and MedCPT retriever, Discuss-RAG yields consistent accuracy gains over MedRAG (up to +16.67% on BioASQ and +12.20% on PubMedQA) and improves snippet quality. The approach is modular and training-free, enhancing reliability and grounding in medical evidence, which can reduce hallucinations and improve practical deployment in clinical QA settings.

Abstract

Medical question answering (QA) is a reasoning-intensive task that remains challenging for large language models (LLMs) due to hallucinations and outdated domain knowledge. Retrieval-Augmented Generation (RAG) provides a promising post-training solution by leveraging external knowledge. However, existing medical RAG systems suffer from two key limitations: (1) a lack of modeling for human-like reasoning behaviors during information retrieval, and (2) reliance on suboptimal medical corpora, which often results in the retrieval of irrelevant or noisy snippets. To overcome these challenges, we propose Discuss-RAG, a plug-and-play module designed to enhance the medical QA RAG system through collaborative agent-based reasoning. Our method introduces a summarizer agent that orchestrates a team of medical experts to emulate multi-turn brainstorming, thereby improving the relevance of retrieved content. Additionally, a decision-making agent evaluates the retrieved snippets before their final integration. Experimental results on four benchmark medical QA datasets show that Discuss-RAG consistently outperforms MedRAG, especially significantly improving answer accuracy by up to 16.67% on BioASQ and 12.20% on PubMedQA. The code is available at: https://github.com/LLM-VLM-GSL/Discuss-RAG.

Figures (4)

• Figure 1: The illustration of difference between RAG and human for a medical query.
• Figure 2: Preliminary experiments on the MMLU-Med benchmark. (A). Accuracy trends as the number of retrieved documents k varies. Three representative questions ($Q_1,Q_2$ and $Q_3$) are selected to illustrate. (B). Examples of retrieved snippets and the corresponding LLM (e.g., GPT-3.5) responses. (C). Example of agent-led snippet selection and the resulting response for query $Q_3$. Additional details are discussed in Sec. \ref{['Sec:pre']}.
• Figure 3: Illustration of the Discuss-RAG pipeline. (A). depicts the multi-turn brainstorming and summarization process. (B). presents the agent-led post-retrieval verification module. A medical query, the corresponding snippets, and the LLM’s generated answer are used for illustration. Further details are provided in Sec \ref{['Sec:method']}.
• Figure 4: Example from the MedQA-US benchmark comparing MedRAG (A) and Discuss-RAG (B). Answers and key phrases are highlighted in red.