Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

Abstract

Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis. Through systematic empirical investigation, we identify that the imbalanced attention allocation, both across modalities (i.e., vision and language) and within modalities (among individual tokens), exhibits a strong causal correlation with the occurrence of object hallucination. Leveraging this insight, we introduce a novel concept termed attention imbalance, which not only quantifies the degree of attention disparity but also visually delineates the underlying patterns (e.g., over-attentiveness to irrelevant language tokens or under-attentiveness to discriminative visual features) that drive object hallucination. To mitigate object hallucination, we further propose Attention Imbalance Rectification (AIR), a lightweight decoding-time intervention method that reallocates attention weights and adjusts attention distributions to rectify modality-wise and token-wise imbalances. Extensive evaluations on four mainstream LVLMs and three benchmarks (CHAIR, POPE, and MM-Vet) with seven baselines demonstrate that AIR consistently reduces object hallucination rates, achieving up to a 35.1% reduction compared to the baselines, while improving up to 15.9% of LVLMs' general capability across diverse vision-language tasks.

Figures (10)

• Figure 1: Attention imbalance causes LLaVA-1.5 to hallucinate (up) while hallucination disappears once the attention is balanced by our AIR approach (down).
• Figure 2: The attention map of output tokens. Imbalanced token is highlighted with a red box, and the hallucinated tokens are indicated by red arrows $\downarrow$.
• Figure 3: Co-occurrence between imbalanced tokens and object hallucinations across LVLMs.
• Figure 4: (a) The distribution of attention heads. We highlight the top twenty hallucination-sensitive heads with red boxes and the bottom twenty hallucination-insensitive heads with blue boxes. (b) Comparison of the average modality-token attention weights among hallucination-sensitive heads, average heads, and hallucination-insensitive heads. (c) Comparison of the total modality-level attention weights of three head groups. (d) Comparison of attention map similarities between the LVLM and its base language model.
• Figure 5: POPE hallucination evaluation results of four LVLMs across different baselines. The four filled regions denote Adversarial, Popular, Random subsets and their average. AIR consistently achieves the highest average performance across all four LVLMs.