Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models

TL;DR

The paper tackles the problem of multimodal hallucination snowballing in large vision-language models by introducing MMHalSnowball, a framework that creates hallucinatory visual conversations to test whether prior hallucinations bias subsequent responses. It shows a substantial performance drop ($\geq 31\%$) across open-source LVLMs under hallucinatory context and introduces Residual Visual Decoding (RVD), a training-free decoding method that emphasizes visual information to mitigate snowballing by over $24\%$ while preserving contextual abilities. The approach combines a careful data-collection pipeline, diverse evaluation metrics, and cross-model analysis, highlighting the fragility of current LVLMs to accumulated hallucinations and the potential of RVD to bolster reliability. The work also introduces adaptive distribution blending guided by distribution similarity and provides auxiliary tasks (WPI) to assess contextual access, offering a practical, model-agnostic mitigation that can be integrated into multi-turn visual dialogue systems.

Abstract

Though advanced in understanding visual information with human languages, Large Vision-Language Models (LVLMs) still suffer from multimodal hallucinations. A natural concern is that during multimodal interaction, the generated hallucinations could influence the LVLMs' subsequent generation. Thus, we raise a question: When presented with a query relevant to the previously generated hallucination, will LVLMs be misled and respond incorrectly, even though the ground visual information exists? To answer this, we propose a framework called MMHalSnowball to evaluate LVLMs' behaviors when encountering generated hallucinations, where LVLMs are required to answer specific visual questions within a curated hallucinatory conversation. Crucially, our experiment shows that the performance of open-source LVLMs drops by at least $31\%$, indicating that LVLMs are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions. We term this phenomenon Multimodal Hallucination Snowballing. To mitigate this, we further propose a training-free method called Residual Visual Decoding, where we revise the output distribution of LVLMs with the one derived from the residual visual input, providing models with direct access to the visual information. Experiments show that our method can mitigate more than $24\%$ of the snowballed multimodal hallucination while maintaining capabilities.

Figures (21)

• Figure 1: An example of the LVLM assisting a visually impaired person to cross the street. The model is misled by the generated hallucination and mistakenly suggests the user to cross the street, although it can give correct advice independently. Green and red colors highlight the correct answer and hallucinations, respectively.
• Figure 2: Preliminary explorations on the hallucinations generated by LVLMs given conversational contexts. (a) Response accuracy with or without hallucinatory conversation. (b) Response distribution when asking the question within a hallucinatory conversation. We select question samples that the LVLM can correctly answer without distractions.
• Figure 3: An overview of our MMHalSnowball framework for simulating hallucinatory conversations and evaluating LVLMs' behavior in such conversations. In step 1, start with a question-answer pair, we generate a fact, an image description and allocate a proper hallucination type according to the corresponding question-answer pair. In step 2, we utilize the ChatGPT to rewrite a hallucinatory answer based on the allocated hallucination type. We then modify other annotations and generate the corresponding hallucinatory description using ChatGPT. In step 3, after ensuring the hallucinatory answer and descriptions contradict the image content, we construct a conversation that contains the specific hallucination. In step 4, we evaluate the LVLMs' performance gap in two conversation settings to see whether they suffer from multimodal hallucination snowballing. Green and red color highlight the correct answer and hallucinations curated out of it, respectively.
• Figure 4: Statistics of our curated dataset.
• Figure 5: Question answering accuracy(a) and flip rate(b) of two different context settings (i.e. HalluConv. and CleanConv.) for each hallucination type. Note that the stripe pattern represents a performance drop due to the snowballed hallucination.