MedHallBench: A New Benchmark for Assessing Hallucination in Medical Large Language Models

TL;DR

MedHallBench tackles the critical issue of hallucinations in medical LLMs by introducing an automated, expert-anchored benchmark that combines textual case scenarios with medical databases and a novel ACHMI metric for medical image-caption hallucination. The framework integrates active learning and RLHF-driven automatic annotation, supported by a PPO-based objective, to enable scalable, clinically reliable evaluation. Empirical results across diverse LLMs demonstrate that ACHMI provides a more nuanced view of hallucination risk than traditional metrics, enabling targeted improvements for safer medical AI deployment. This work lays a foundation for reliable, domain-specific evaluation and mitigation of AI hallucinations in healthcare contexts.

Abstract

Medical Large Language Models (MLLMs) have demonstrated potential in healthcare applications, yet their propensity for hallucinations -- generating medically implausible or inaccurate information -- presents substantial risks to patient care. This paper introduces MedHallBench, a comprehensive benchmark framework for evaluating and mitigating hallucinations in MLLMs. Our methodology integrates expert-validated medical case scenarios with established medical databases to create a robust evaluation dataset. The framework employs a sophisticated measurement system that combines automated ACHMI (Automatic Caption Hallucination Measurement in Medical Imaging) scoring with rigorous clinical expert evaluations and utilizes reinforcement learning methods to achieve automatic annotation. Through an optimized reinforcement learning from human feedback (RLHF) training pipeline specifically designed for medical applications, MedHallBench enables thorough evaluation of MLLMs across diverse clinical contexts while maintaining stringent accuracy standards. We conducted comparative experiments involving various models, utilizing the benchmark to establish a baseline for widely adopted large language models (LLMs). Our findings indicate that ACHMI provides a more nuanced understanding of the effects of hallucinations compared to traditional metrics, thereby highlighting its advantages in hallucination assessment. This research establishes a foundational framework for enhancing MLLMs' reliability in healthcare settings and presents actionable strategies for addressing the critical challenge of AI hallucinations in medical applications.

Figures (4)

• Figure 1: The figure highlights that LLaVA1.5-7b, LLaVA1.5-13b, and MiniGPT-4 are all unable to correctly describe X-ray images, showing a significant deviation from the actual doctor's response. This illustrates the hallucination problem of LLMs in the medical field, where the models fail to provide accurate and reliable interpretations.
• Figure 2: Methodological Framework for Medical Dataset Construction and Validation.
• Figure 3: Iterative Pipeline for Hallucination Detection and Correction in Multimodal Medical Datasets. The process of manual annotation in hallucination detection and correction in multimodal datasets begins with SFT Data, using an expert question and answer format. This data then proceeds to automatic annotation, involving a feedback loop with experts and ChatGLM. Subsequently, a user study is conducted to further refine the process. Following the user study, annotation data exhibiting low confidence is identified. These data are then selected for active learning, leading to manual correction and subsequent model retraining.
• Figure 4: RLHF Training Pipeline for Medical Language Models.