Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding

TL;DR

LVLMs suffer object-level hallucinations from both visual perception limits and language priors. The authors propose Hallucination Disentangled Decoding (HDD), a training-free method that uses semantic segmentation to amplify local image details and blank-image contrastive decoding to suppress language priors, selecting the most informative segmentation via Jensen-Shannon divergence and adaptively fusing its signal with the original. HDD achieves state-of-the-art performance across multiple benchmarks (POPE, CHAIR, GPT-4 benchmark) and datasets, reducing hallucinations while preserving descriptive detail. The approach is robust to segmentation choices and maintains practical efficiency, highlighting its potential for more reliable multimodal reasoning in real-world applications.

Abstract

Large Vision-Language Models (LVLMs) bridge the gap between visual and linguistic modalities, demonstrating strong potential across a variety of domains. However, despite significant progress, LVLMs still suffer from severe hallucination issues in object recognition tasks. These models often fail to accurately identify certain objects, leading to text generation that appears fluent but does not correspond to the visual content, which can have serious consequences in real-world applications. Recently, several methods have been proposed to alleviate LVLM hallucinations, but most focus solely on reducing hallucinations in the language modality. To mitigate hallucinations in both the language and visual modalities, we introduce Hallucination Disentangled Decoding (HDD) method that requires no training. HDD enhances the original image by segmenting it and selecting images that augment the original, while also utilizing a blank image to eliminate language prior hallucinations in both the original and segmented images. This design not only reduces the model's dependence on language priors but also enhances its visual performance. (Code: https://github.com/rickeyhhh/Hallucination-Disentangled-Decoding)

Figures (9)

• Figure 1: The figure above illustrates the impact of different object sizes in the image on the model's output logits distribution. From top to bottom, the area of the "car" gradually decreases, while the area of the "person" remains unchanged, serving as a control.
• Figure 2: The figure above demonstrates the impact of different prompt inputs on the model's output. It explains the phenomenon of "Decoding Inertia" by introducing a blank image and two distinct prompt inputs.
• Figure 3: This figure provides an overview of Hallucination Disentangled Decoding. First, the original image is segmented into two images. The blank image, along with the three aforementioned images, is then input into the LVLM, and their respective output distributions are obtained. By calculating the Jensen-Shannon (JS) divergence between the distributions of the two images and the blank image, the image with the higher JS divergence is selected. The original image and the chosen image are then subtracted from the blank image's distribution and summed to obtain the new output distribution.
• Figure 4: GPT-4 assisted hallucination evaluation results on VG dataset. We analyse the results across six dimensions, where HWR, HSR, HSPI, and HWPI are better when lower, while WPI and SPI are better when higher. Overall, a larger proportion on the radar chart indicates better performance.
• Figure 5: The decoding differences on the MSCOCO dataset are presented, where the Red text highlights hallucinated regions, and the Green text represents additional detailed descriptions not present in the original decoding. Our method not only reduces hallucinated information but also provides more detailed descriptions.