SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models

TL;DR

This work identifies object hallucination in large vision-language models as stemming from a texture-biased Bag-of-Patches behavior in vision encoders, which undermines reliance on global structural cues. It introduces Structure-Disrupted Contrastive Decoding (SDCD), a training-free inference-time calibration that contrasts the original visual view with a structure-disrupted shuffled view to penalize texture-driven tokens via a contrastive logit combination. Empirical results across POPE, MME, and CHAIR show that SDCD substantially reduces hallucinations while preserving or improving multimodal reasoning, with architecture-dependent gains favoring patch-wise projection designs. The method highlights the critical role of visual structure in LVLM decoding and provides a generally applicable, efficient approach to mitigating hallucinations in diverse model architectures.

Abstract

Large Vision-Language Models (LVLMs) demonstrate significant progress in multimodal understanding and reasoning, yet object hallucination remains a critical challenge. While existing research focuses on mitigating language priors or high-level statistical biases, they often overlook the internal complexities of the visual encoding process. We identify that visual statistical bias, arising from the inherent Bag-of-Patches behavior of Vision Encoders under weak structural supervision, acts as a contributing factor of object hallucinations. Under this bias, models prioritize local texture features within individual patches over holistic geometric structures. This tendency may induce spurious visual confidence and result in hallucinations. To address this, we introduce a training-free algorithm called Structure-Disrupted Contrastive Decoding (SDCD), which performs contrastive calibration of the output distribution by introducing a shuffled structure-disrupted view. By penalizing tokens that maintain high confidence under this structure-less view, SDCD effectively suppresses the texture-driven bias. Experimental results demonstrate that SDCD significantly mitigates hallucinations across multiple benchmarks and enhances the overall multimodal capabilities of LVLMs.

Figures (3)

• Figure 1: Overview of the proposed SDCD framework. Given a textual input $x$, the model performs contrastive decoding by jointly leveraging the original view $V$, a shuffled view $V'$. By suppressing texture-driven bias, SDCD effectively eliminates hallucinated objects during generation.
• Figure 2: Quantitative analysis of the Bag-of-Patches behavior. (a) Top-1 Accuracy of ViT-B/16 on a random subset of 500 images from the ImageNet validation set under varying patch shuffling granularities. (b) Image-Text Retrieval performance (R@1 and R@5) of CLIP-ViT-L/14. We extract visual features from original and shuffled views and compute cosine similarity against text features from 5 corresponding captions per image. Despite severe structural disruption, both models retain strong semantic performance.
• Figure 3: An illustration of Structure Sensitivity Divergence in LVLM decoding. We analyze the logit dynamics of Yes and No tokens under the original view ($V$) and the shuffled view ($V'$). (a) For ground-truth No cases (potential hallucinations), the confidence of the incorrect Yes token often increases under $V'$, indicating a texture-dominated response when global structure is removed. (b) For ground-truth Yes cases (real objects), the confidence of the correct Yes token drops sharply under $V'$, revealing a strong dependence on global geometric structure (structural penalty). The shaded regions highlight the difference in token logits between the original view ($V$) and the shuffled view ($V'$). This asymmetric response characterizes the Structure Sensitivity Divergence between real and hallucinated objects.