Reducing Hallucinations of Medical Multimodal Large Language Models with Visual Retrieval-Augmented Generation

TL;DR

This work tackles hallucinations in Medical Multimodal LLMs by introducing Visual RAG (V-RAG), which grounds generation using retrieved both medical images and their accompanying texts. It combines a multimodal retrieval pipeline based on BiomedCLIP and FAISS/HNSW with a prompting strategy and a set of fine-tuning tasks to enable V-RAG on both multi-image and single-image Med-MLLMs. Through entity probing on MIMIC-CXR and MultiCaRe, and subsequent report rewriting via a senior radiologist prompt, V-RAG improves grounding for both frequent and rare entities and increases RadGraph-F1 for chest X-ray reports. The results demonstrate that multimodal retrieval and targeted fine-tuning enable more trustworthy clinical outputs, paving the way for safer deployment of Med-MLLMs in radiology and broader medical imaging tasks.

Abstract

Multimodal Large Language Models (MLLMs) have shown impressive performance in vision and text tasks. However, hallucination remains a major challenge, especially in fields like healthcare where details are critical. In this work, we show how MLLMs may be enhanced to support Visual RAG (V-RAG), a retrieval-augmented generation framework that incorporates both text and visual data from retrieved images. On the MIMIC-CXR chest X-ray report generation and Multicare medical image caption generation datasets, we show that Visual RAG improves the accuracy of entity probing, which asks whether a medical entities is grounded by an image. We show that the improvements extend both to frequent and rare entities, the latter of which may have less positive training data. Downstream, we apply V-RAG with entity probing to correct hallucinations and generate more clinically accurate X-ray reports, obtaining a higher RadGraph-F1 score.

Figures (4)

• Figure 1: (Up) Hallucination issue of Med-MLLM. (Down) Framework of V-RAG to improve Med-MLLM.
• Figure 2: Entity probing asks entity-based questions to an MLLM and compares predictions against answers grounded in an LLM's interpretation of a reference caption.
• Figure 3: Fine-tuning tasks to make Med-MLLM V-RAG-capable by (a) improving image-and-text association abilities, (b) focusing on specific images, and (c) making decisions using extracted similar data.
• Figure 4: F1 performance of each method on disease entities with different frequencies. The superior performance of our method, particularly in probing rare entities, demonstrates its effectiveness and applicability in real-world scenarios.