Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding

TL;DR

The paper addresses object hallucination in large vision-language models by introducing ATED, a training-free token-level ensemble that aggregates per-token logits from multiple LVLMs during decoding using uncertainty-guided weights and multi-path contrastive decoding. ATED operates at inference time, dynamically weighting model contributions to improve factual grounding and semantic consistency without retraining. Extensive experiments on POPE, CHAIR, and MME show that ATED consistently outperforms single-model baselines and existing decoding strategies, achieving substantial gains in accuracy and robustness while offering flexible latency-accuracy trade-offs. This work demonstrates the efficacy of adaptive ensemble decoding for enhancing LVLM reliability in high-stakes multimodal tasks and provides a scalable, model-agnostic approach to mitigate hallucinations.

Abstract

Large Vision-Language Models (LVLMs) have recently achieved impressive results in multimodal tasks such as image captioning and visual question answering. However, they remain prone to object hallucination -- generating descriptions of nonexistent or misidentified objects. Prior work has partially mitigated this via auxiliary training objectives or external modules, but challenges remain in terms of scalability, adaptability, and model independence. To address these limitations, we propose Adaptive Token Ensemble Decoding (ATED), a training-free, token-level ensemble framework that mitigates hallucination by aggregating predictions from multiple LVLMs during inference. ATED dynamically computes uncertainty-based weights for each model, reflecting their reliability at each decoding step. It also integrates diverse decoding paths to improve contextual grounding and semantic consistency. Experiments on standard hallucination detection benchmarks demonstrate that ATED significantly outperforms state-of-the-art methods, reducing hallucination without compromising fluency or relevance. Our findings highlight the benefits of adaptive ensembling and point to a promising direction for improving LVLM robustness in high-stakes applications. The code is available at https://github.com/jinlin2021/ATED.

Figures (10)

• Figure 1: Comparison of image‐description generation results from various LVLMs and our proposed ATED method. Red text indicates hallucination. Green text represents hallucination mitigating from ATED.
• Figure 2: Overall pipeline of Adapative Token Ensemble Decoding (ATED). In module (a), the system takes as input a set of system instructions, the original image, and its perturbed variants, which are passed to multiple candidate LVLMs in module (b). Module (c) generates multiple decoding streams. In module (d), the importance of each LVLM’s output is estimated using uncertainty-guided weights and refined via greedy uncertainty optimization. Finally, module (e) aggregates the results through ensemble decoding to produce the final output. This process is dynamically repeated at each time step $t$ during token generation, ensuring both reliability and consistency of the results.
• Figure 3: Results on the hallucination subset of MME with decoding strategies: LLaVA-1.5 (top), InstructBLIP (bottom).
• Figure 4: Trade-off between inference latency and caption generation performance.
• Figure 5: GPT-4 assisted hallucination evaluation.