Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

TL;DR

LVLMs enable clinical visual-language tasks but are vulnerable to medical hallucinations. The authors introduce Med-HallMark, a medical multimodal benchmark with multi-task support, multifaceted data, and a five-level hierarchy. They also propose MediHall Score, a hierarchical metric, and MediHallDetector, a specialized detector trained with multi-source data. Experiments show MediHall Score provides nuanced assessment beyond traditional metrics, and MediHallDetector improves detection efficiency and consistency, advancing reliability of medical LVLMs.

Abstract

Large Vision Language Models (LVLMs) are increasingly integral to healthcare applications, including medical visual question answering and imaging report generation. While these models inherit the robust capabilities of foundational Large Language Models (LLMs), they also inherit susceptibility to hallucinations-a significant concern in high-stakes medical contexts where the margin for error is minimal. However, currently, there are no dedicated methods or benchmarks for hallucination detection and evaluation in the medical field. To bridge this gap, we introduce Med-HallMark, the first benchmark specifically designed for hallucination detection and evaluation within the medical multimodal domain. This benchmark provides multi-tasking hallucination support, multifaceted hallucination data, and hierarchical hallucination categorization. Furthermore, we propose the MediHall Score, a new medical evaluative metric designed to assess LVLMs' hallucinations through a hierarchical scoring system that considers the severity and type of hallucination, thereby enabling a granular assessment of potential clinical impacts. We also present MediHallDetector, a novel Medical LVLM engineered for precise hallucination detection, which employs multitask training for hallucination detection. Through extensive experimental evaluations, we establish baselines for popular LVLMs using our benchmark. The findings indicate that MediHall Score provides a more nuanced understanding of hallucination impacts compared to traditional metrics and demonstrate the enhanced performance of MediHallDetector. We hope this work can significantly improve the reliability of LVLMs in medical applications. All resources of this work will be released soon.

Figures (14)

• Figure 1: Illustration of statistical information and construction content of Med-HallMark. We show separately (a) multi-task hallucination support, (b) multifaceted hallucination data, and (c) hierarchical hallucination categorization.
• Figure 2: Visualization of MediHalldetector related information. (a) Model structure, SFT process and inference objective of MediHalldetector. (b) Examples of questions, LVLM answers and $GT$ for different types of tasks. (c) Comparison of three rounds of evaluation agreement and average inference time for different evaluation models. (d) Comparison of different evaluation models' agreement with human evaluation preferences in different hallucination texts.
• Figure 3: Comparison of different Models on hallucination types.
• Figure 4: Prefix of confidence-weakening questions.
• Figure 5: Prompts for GPT-4 to create counterfactual questions.