MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models

TL;DR

MedHallu introduces the first medical-hallucination benchmark designed to detect factually incorrect outputs in medical QA. It builds a 10,000-sample dataset from PubMedQA with a controlled hallucination-generation pipeline, multi-model discriminative filtering, and bidirectional-entailment-based analysis, further augmented by a not-sure abstention option and domain knowledge integration. Key findings show that hallucinations closer to ground truth are harder to detect, domain knowledge broadly improves detection, and general LLMs can outperform medical-fine-tuned models in many settings. The benchmark provides a scalable, domain-specific resource to evaluate LLM safety in high-stakes medical contexts and informs strategies for safer deployment and future research.

Abstract

Advancements in Large Language Models (LLMs) and their increasing use in medical question-answering necessitate rigorous evaluation of their reliability. A critical challenge lies in hallucination, where models generate plausible yet factually incorrect outputs. In the medical domain, this poses serious risks to patient safety and clinical decision-making. To address this, we introduce MedHallu, the first benchmark specifically designed for medical hallucination detection. MedHallu comprises 10,000 high-quality question-answer pairs derived from PubMedQA, with hallucinated answers systematically generated through a controlled pipeline. Our experiments show that state-of-the-art LLMs, including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical, struggle with this binary hallucination detection task, with the best model achieving an F1 score as low as 0.625 for detecting "hard" category hallucinations. Using bidirectional entailment clustering, we show that harder-to-detect hallucinations are semantically closer to ground truth. Through experiments, we also show incorporating domain-specific knowledge and introducing a "not sure" category as one of the answer categories improves the precision and F1 scores by up to 38% relative to baselines.

Figures (7)

• Figure 1: An example of medical hallucination detection. The detailed prompt used for the hallucination detection task is presented in Appendix \ref{['appendix:prompt']}.
• Figure 2: MedHallu medical hallucinated answer generation pipeline. Each question-answer pair from the PubMedQA dataset undergoes the following steps to generate a hallucinated answer: (1) Candidate Generation: Given a question, relevant knowledge, and ground truth answer, the LLM is prompted to generate a hallucinated answer adhering to one of four hallucination types. (2) Grading & Filtering: Generated answers undergo quality and correctness checks, being labeled as hard, medium, easy, or failed based on filtering results. (3) Refining Failed Generation: Failed answers are optimized using TextGrad yuksekgonul2024textgradautomaticdifferentiationtext and re-filtered. If they fail again, the LLM is re-prompted to generate new answers (Regeneration). (4) Fallback: If no qualified answers emerge after four regeneration attempts, the answer most similar to the ground truth is selected as an easy hallucinated example. The detailed prompt used for hallucination generation task is presented in the Appendix \ref{['appendix:prompt']}.
• Figure 3:
• Figure 4:
• Figure 5: