Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense

Yiyang Huang, Liang Shi, Yitian Zhang, Yi Xu, Yun Fu

TL;DR

This work identifies object hallucination in LVLMs as arising from visual encoder biases: statistical bias, inherent bias, and vulnerability. It introduces SHIELD, a training-free framework that combines token re-weighting to reduce statistical bias, token subtraction to mitigate inherent bias, and contrastive decoding to counter vulnerability. Across multiple LVLMs and benchmarks (CHAIR, POPE, MME, LLaVA-Bench), SHIELD consistently lowers hallucination incidence and improves broad perception capabilities, with only moderate inference overhead. By focusing on the visual encoder rather than solely on the LLM, SHIELD offers a practical, broadly applicable approach to safer, more reliable cross-modal reasoning in LVLMs.

Abstract

Large Vision-Language Models (LVLMs) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains a significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucinations to visual encoders and identifies three key issues: statistical bias, inherent bias, and vulnerability. To address these challenges, we propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD effectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code will be released.

SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense

TL;DR

This work identifies object hallucination in LVLMs as arising from visual encoder biases: statistical bias, inherent bias, and vulnerability. It introduces SHIELD, a training-free framework that combines token re-weighting to reduce statistical bias, token subtraction to mitigate inherent bias, and contrastive decoding to counter vulnerability. Across multiple LVLMs and benchmarks (CHAIR, POPE, MME, LLaVA-Bench), SHIELD consistently lowers hallucination incidence and improves broad perception capabilities, with only moderate inference overhead. By focusing on the visual encoder rather than solely on the LLM, SHIELD offers a practical, broadly applicable approach to safer, more reliable cross-modal reasoning in LVLMs.

Abstract

Large Vision-Language Models (LVLMs) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains a significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucinations to visual encoders and identifies three key issues: statistical bias, inherent bias, and vulnerability. To address these challenges, we propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD effectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code will be released.
Paper Structure (30 sections, 17 equations, 9 figures, 14 tables)

This paper contains 30 sections, 17 equations, 9 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Large Vision Language Models (LVLMs)
  4. Object Hallucination in LVLMs
  5. Method
  6. Hallucinations Stem from Visual Encoder
  7. Statistical and Inherent Bias in Visual Encoder
  8. Vulnerability in Visual Encoder
  9. SHIELD: suppressing hallucinations in lvlm encoders via bias and vulnerability defense
  10. Formulation of LVLM Inference
  11. Mitigating Statistical Bias
  12. Reducing Inherent Bias
  13. Address Vulnerability
  14. Experiments
  15. Implementation Details
  16. ...and 15 more sections

Figures (9)

  • Figure 1: Key issues underlying object hallucinations in LVLMs. Statistical bias: the visual encoder overemphasizes frequent visual patterns, distorting fine-grained perception. Inherent bias: the encoder produces erroneous representations of dominant objects in the pretraining data, regardless of input. Vulnerability: the encoder is sensitive to minor perturbations, yielding unreliable features.
  • Figure 2: Statistics show that hallucinations stem from bias and vulnerability. (a) The X-axis shows the peak-to-average L2 norm ratio of visual tokens, measuring the deviation of the highest-norm token from the average, and the Y-axis shows the proportion of hallucinating samples at each level. Stronger overemphasis leads to higher hallucination rates. (b) The X-axis lists objects, and the Y-axis shows hallucination occurrences under meaningless inputs. Dominant objects are more likely to be falsely perceived as present. (c) The X-axis denotes the number of attack steps, and the Y-axis shows the F1 score. Even small perturbations increase hallucinations and degrade performance.
  • Figure 3: Illustration of the proposed SHIELD framework. Given an input image and a query text, the visual encoder produces tokens affected by statistical bias (overemphasized tokens ), inherent bias (erroneous representations ), and vulnerability (inaccurate features ). SHIELD addresses these issues through three modules: (i) Token Re-weighting, which redistributes attention to more ground-truth-object relevant tokens to alleviate overemphasis (); (ii) Token Subtraction, which estimates and removes erroneous representations () via noise-derived tokens; and (iii) Contrastive Decoding, which exposes inaccurate features () using attacked images and suppresses corresponding outputs by contrasting them with those from the natural image.
  • Figure 4: Mitigating Statistical Bias. Visual tokens are re-weighted via a similarity matrix between visual tokens and naive caption tokens, emphasizing more ground-truth-object relevant tokens and reducing overemphasis.
  • Figure 5: Reducing Inherent Bias. $K$ noise inputs are used to estimate erroneous representations of dominant objects in the pretraining data, which are then removed from visual tokens via feature subtraction.
  • ...and 4 more figures