Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models

TL;DR

Hallucinogen introduces a novel benchmark to systematically evaluate hallucinations in large vision-language models via implicit contextual reasoning attacks. It contrasts explicit prompts with implicit, context-driven prompts across salient and latent visual entities, quantified on a large dataset of $90{,}000$ triplets (and per-image prompts), across 11 LVLMs. Results show widespread vulnerability: implicit attacks yield higher hallucination than explicit ones, latent-entity prompts drive near-random performance in medical imagery, and existing defenses (post-prompt, CoT, RLHF, LURE) offer limited protection. The study highlights a critical gap in visual grounding within LVLMs and points to the need for robust, grounding-based mitigation strategies to ensure reliable multimodal reasoning in high-stakes settings.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks. However, these models still suffer from hallucinations, particularly when required to implicitly recognize or infer diverse visual entities from images for complex vision-language tasks. To address this challenge, we propose HALLUCINOGEN, a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks to evaluate the extent of hallucination in state-of-the-art LVLMs. Our benchmark provides a comprehensive study of the implicit reasoning capabilities of these models by first categorizing visual entities based on the ease of recognition in an image as either salient (prominent, visibly recognizable objects such as a car) or latent entities (such as identifying a disease from a chest X-ray), which are not readily visible and require domain knowledge or contextual reasoning for accurate inference. Next, we design hallucination attacks for both types of entities to assess hallucinations in LVLMs while performing various vision-language tasks, such as locating or reasoning about specific entities within an image, where models must perform implicit reasoning by verifying the existence of the queried entity within the image before generating responses. Finally, our extensive evaluations of eleven LVLMs, including powerful open-source models (like LLaMA-3.2 and DeepSeek-V2), commercial models like Gemini, and two hallucination mitigation strategies across multiple datasets, demonstrate that current LVLMs remain susceptible to hallucination attacks.

Figures (8)

• Figure 1: Examples of different object hallucination attacks, where hallucination prompts from Hallucinogen (right) are able to make the LVLM hallucinate response. (Left) When explicitly asked to identify a non-existent object, such as "person," LVLMs like LLaVA1.5 liu2024improved generate a correct response. (Right) However, in the case of an implicit object hallucination attack, where the question requires first implicitly determining an object's presence before describing its position, the LVLMs produce a hallucinated response.
• Figure 2: Illustration of various types of hallucination attacks in Hallucinogen. We broadly define two categories of hallucination attacks: explicit and implicit attacks. An explicit attack involves directly prompting LVLMs to accurately identify the presence or absence of existing or non-existing visual entity. In contrast, an implicit attack employs more complex queries that do not explicitly inquire about a specific visual entity but instead require the model to implicitly assess its presence in the image to generate a factually accurate response. Furthermore, for implicit attacks, we propose a range of visual-language tasks with varying levels of difficulty, from correctly locating a visual entity to understanding its surrounding context.
• Figure 3: We benchmark eleven state-of-the-art LVLMs on the Hallucinogen. Using image-entity pairs categorized as (top) salient and (bottom) latent entities, we evaluate these LVLMs across diverse tasks, including Localization (LOC), Visual Context (VC), and Counterfactual reasoning (CF). Lower accuracy reflects incorrectness in inferring the presence or absence of an object, which correlates with a higher degree of object hallucination.
• Figure 4: Comparing attention scores for visual, query, and previously generated tokens while predicting the next tokens. The (left) plot illustrates the trend in attention scores for localization tasks, while the (right) plot depicts the trend for counterfactual reasoning tasks. Overall, we observe that LVLMs allocate very little attention to visual tokens when responding to our hallucination attacks.
• Figure 5: Error Analysis on the incorrect responses generated by Qwen2VL yang2024qwen2 when evaluated across Hallucinogen attack on diverse vision-language tasks.