Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context

TL;DR

The paper addresses why LVLMs are more prone to hallucinations in longer, free-form responses, challenging the notion that length alone drives errors. It shows that context—specifically coherence and completeness—plays a central role in hallucination risk, with evidence that hallucinations can arise earlier as richer context is provided. Building on this, the authors introduce HalTrapper, a zero-shot induce-detect-suppress framework that first induces potential hallucinations, then detects them via internal grounding and external expansion signals, and finally suppresses them through Contrastive Contextual Decoding. Across multiple benchmarks and models, HalTrapper delivers consistent improvements in hallucination mitigation without additional training, offering new insights into the mechanism of LVLM hallucinations and a practical tool for long-form multimodal generation.

Abstract

Large Vision-Language Models (LVLMs) have made significant progress in recent years but are also prone to hallucination issues. They exhibit more hallucinations in longer, free-form responses, often attributed to accumulated uncertainties. In this paper, we ask: Does increased hallucination result solely from length-induced errors, or is there a deeper underlying mechanism? After a series of preliminary experiments and findings, we suggest that the risk of hallucinations is not caused by length itself but by the increased reliance on context for coherence and completeness in longer responses. Building on these insights, we propose a novel "induce-detect-suppress" framework that actively induces hallucinations through deliberately designed contexts, leverages induced instances for early detection of high-risk cases, and ultimately suppresses potential object-level hallucinations during actual decoding. Our approach achieves consistent, significant improvements across all benchmarks, demonstrating its efficacy. The strong detection and improved hallucination mitigation not only validate our framework but, more importantly, re-validate our hypothesis on context. Rather than solely pursuing performance gains, this study aims to provide new insights and serves as a first step toward a deeper exploration of hallucinations in LVLMs' longer responses.

Figures (14)

• Figure 1: Left: Our three main findings and the three steps of our HalTrapper method. Right: The distribution of hallucination locations detected by our HalTrapper is close to the true distribution of hallucinations, indicating that our method, to some extent, captures the essence of LVLM hallucinations.
• Figure 2: Statistical analysis of hallucination positions under context modifications. Both cropping the image and enriching the prompt lead to earlier hallucination occurrences.
• Figure 3: Statistical analysis related to contextual coherence. Within the same caption, hallucinated object pairs exhibit higher attention similarity scores than non-hallucinated pairs.
• Figure 4: Statistical analysis related to contextual completeness: (a) Hallucination positions shift progressively earlier as more image information is included in the prompts. (b) Similar hallucinations consistently recur across varied prompts for the same image.
• Figure 5: Overview of HalTrapper: It consists of two branches leveraging coherence and completeness insights. One generates captions with an appended "There is also" prompt to induce potential hallucinated objects, detected via high attention similarity between caption and induced tokens. The other prompts the LVLM to imagine surrounded content beyond the image to identify consistent hallucinations. With detected hallucinated objects, HalTrapper further suppresses hallucinations through Contrastive Contextual Decoding.