Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

TL;DR

This work tackles object hallucination in large vision-language models by introducing Ensemble Decoding, which splits an input image into sub-images and dynamically ensembles their logits via attention-guided weights, along with an ED adaptive plausibility constraint. A faster variant, FastED, reduces computation by relying on the most informative sub-image, balancing performance and speed. Across POPE, CHAIR, MME, and LLaVA-Bench benchmarks, ED achieves state-of-the-art results, improving both existence-related accuracy and caption detail, while qualitative analyses show more accurate, dynamically guided attention. The method is module-free and training-free, leveraging intrinsic visual information through attention, though it currently requires patch-wise locality and lacks a formal theoretical proof. Overall, ED offers practical gains for reducing multimodal hallucinations with scalable applicability to larger LVLMs, with FastED providing a viable option for speed-critical deployments.

Abstract

Recent advancements in Large Vision-Language Models (LVLMs) have significantly expanded their utility in tasks like image captioning and visual question answering. However, they still struggle with object hallucination, where models generate descriptions that inaccurately reflect the visual content by including nonexistent objects or misrepresenting existing ones. While previous methods, such as data augmentation and training-free approaches, strive to tackle this issue, they still encounter scalability challenges and often depend on additional external modules. In this work, we propose Ensemble Decoding (ED), a novel strategy that splits the input image into sub-images and combines logit distributions by assigning weights through the attention map. Furthermore, we introduce ED adaptive plausibility constraint to calibrate logit distribution and FastED, a variant designed for speed-critical applications. Extensive experiments across hallucination benchmarks demonstrate that our proposed method achieves state-of-the-art performance, validating the effectiveness of our approach.

Figures (10)

• Figure 1: Example of object hallucination in LVLM (left). After cropping and resizing the image, the model answers correctly (right).
• Figure 2: Experimental results of the pilot study. Masking refers to masking some irrelevant objects in the image, while Low-Resolution involves reducing the resolution of each object in the image.
• Figure 3: Overall pipeline of Ensemble Decoding (ED). Attention-guided weights are applied to sub-images and combined with the logits from the original image for ensembling. ED adaptive plausibility constraint is applied to generate the final output. The entire process is dynamically repeated at each time step $t$ of token generation.
• Figure 4: Experimental results of MME on a hallucination subset with different decoding strategies.
• Figure 5: Results of GPT-aided evaluation on the captions of LLaVA-Bench.