MedTrust-RAG: Evidence Verification and Trust Alignment for Biomedical Question Answering

TL;DR

Biomedical QA suffers from hallucinations; MedTrust-Guided Iterative RAG mitigates this by enforcing citation-grounded reasoning, an iterative verification loop with Medical Gap Analysis, and a trust-aligned MTAM trained with Direct Preference Optimization. The system grounds every statement to retrieved documents, iteratively refines evidence within $T_{\max}=3$ steps, and uses a rich, hallucination-aware data framework (MedRankQA) to train robust detectors of unreliable content. Empirically, it yields consistent improvements on MedMCQA, MedQA, and MMLU-Med, with average EM gains up to $2.7$ percentage points over strong RAG baselines and clear reductions in multiple hallucination patterns, signaling safer, more interpretable medical AI. This approach advances practical clinical decision support by strengthening factual grounding and accountability in biomedical QA models.

Abstract

Biomedical question answering (QA) requires accurate interpretation of complex medical knowledge. Large language models (LLMs) have shown promising capabilities in this domain, with retrieval-augmented generation (RAG) systems enhancing performance by incorporating external medical literature. However, RAG-based approaches in biomedical QA suffer from hallucinations due to post-retrieval noise and insufficient verification of retrieved evidence, undermining response reliability. We propose MedTrust-Guided Iterative RAG, a framework designed to enhance factual consistency and mitigate hallucinations in medical QA. Our method introduces three key innovations. First, it enforces citation-aware reasoning by requiring all generated content to be explicitly grounded in retrieved medical documents, with structured Negative Knowledge Assertions used when evidence is insufficient. Second, it employs an iterative retrieval-verification process, where a verification agent assesses evidence adequacy and refines queries through Medical Gap Analysis until reliable information is obtained. Third, it integrates the MedTrust-Align Module (MTAM) that combines verified positive examples with hallucination-aware negative samples, leveraging Direct Preference Optimization to reinforce citation-grounded reasoning while penalizing hallucination-prone response patterns.

Figures (4)

• Figure 1: An example illustrating post-retrieval noise in biomedical RAG. The model answers the question correctly without retrieval, but produces an incorrect response when using retrieved documents, highlighting the potential risk of introducing irrelevant or misleading information.
• Figure 2: Illustration of the iterative retrieval-verification pipeline with dual-agent coordination. The verifier agent evaluates citation-grounded reasoning and provides Negative Knowledge Assertions (NKA) and Medical Gap Analysis (GA) for query refinement. The generator produces clinically grounded responses only from verified reasoning.
• Figure 3: Overview of the MedTrust-Align framework for hallucination-aware medical question answering. The left pipeline depicts the knowledge construction process, including difficulty-aware sampling from MedRankQA (MR. QA), hybrid document retrieval, and NLI-based knowledge composition. The right pipeline shows the alignment training process, which involves positive sample generation from validated CiteReason and medically grounded NK. Assertion (Negative Knowledge Assertion), hallucination-aware negative sampling based on reasoning conflicts, and preference optimization through DPO training.
• Figure 4: Distribution of four hallucination types across the MMLU, MedQA and MedMCQA datasets. Subfigures (a)–(d) show Faulty Reasoning, Over Refusal, Missing Answer and Misattribution, respectively. Each bar indicates the proportion under three model variants, where BASE is the original model without alignment, DPO is trained with preference optimization and SFT is trained with supervised fine-tuning.