V-Loop: Visual Logical Loop Verification for Hallucination Detection in Medical Visual Question Answering

TL;DR

This work tackles hallucinations in medical visual question answering (VQA) by introducing Visual Logical Loop Verification (V-Loop), a training-free, plug-and-play framework that performs visual-grounded self-verification. It creates a bidirectional verification loop: the primary VQA answer is examined by generating a verification question conditioned on the answer’s semantic unit, then answered with the same visual evidence under Visual Attention Consistency (VAC), and checked for semantic alignment using a deterministic evaluator. The method supports logic-based verification when dual semantic units exist or rephrase-based verification otherwise, ensuring robust grounding. Experiments across three medical VQA benchmarks and three MLLMs show V-Loop outperforms existing introspective detectors and enhances uncertainty-based methods, with performance improving alongside MLLM competence and remaining robust to the auxiliary LLM used for verification question generation.

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable capability in assisting disease diagnosis in medical visual question answering (VQA). However, their outputs remain vulnerable to hallucinations (i.e., responses that contradict visual facts), posing significant risks in high-stakes medical scenarios. Recent introspective detection methods, particularly uncertainty-based approaches, offer computational efficiency but are fundamentally indirect, as they estimate predictive uncertainty for an image-question pair rather than verifying the factual correctness of a specific answer. To address this limitation, we propose Visual Logical Loop Verification (V-Loop), a training-free and plug-and-play framework for hallucination detection in medical VQA. V-Loop introduces a bidirectional reasoning process that forms a visually grounded logical loop to verify factual correctness. Given an input, the MLLM produces an answer for the primary input pair. V-Loop extracts semantic units from the primary QA pair, generates a verification question by conditioning on the answer unit to re-query the question unit, and enforces visual attention consistency to ensure answering both primary question and verification question rely on the same image evidence. If the verification answer matches the expected semantic content, the logical loop closes, indicating factual grounding; otherwise, the primary answer is flagged as hallucinated. Extensive experiments on multiple medical VQA benchmarks and MLLMs show that V-Loop consistently outperforms existing introspective methods, remains highly efficient, and further boosts uncertainty-based approaches when used in combination.

Figures (6)

• Figure 1: Comparison between uncertainty-based detection method and our proposed V-Loop. (a) Uncertainty-based methods detect hallucinations by sampling multiple answers and estimating semantic entropy. (b) V-Loop instead verifies the factual correctness of a specific answer through visual logical loop verification.
• Figure 2: Overview of the proposed V-Loop framework. It consists of four stages, (a) primary visual question answering, where the MLLM generates the primary answer to the given image-question pair; (b) verification question generation, creating a verification question and its expected answer; (c) verification question answering, where the model answers the verification question under the same visual evidence; and (d) semantic consistency check, which evaluates the consistency between verification answer and its corresponding reference answer.
• Figure 3: Visual Attention Consistency. The upper panel shows the primary stage, and the lower shows the verification process.
• Figure 4: Effect of the weighting factor $\alpha$ on the performance (AUC (%) and AUG (%)) of V-Loop, evaluated on the VQA-RAD open-ended subset with MedGemma-4B-it.
• Figure 5: Examples of V-Loop. In each example, semantic units extracted from the question (blue) and answer (pink) are used to form the verification question. A correct verification answer closes the loop (left), while mismatches indicate hallucination (middle/right).