Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation

Abstract

Despite the significant advancements in Large Vision-Language Models (LVLMs), their tendency to generate hallucinations undermines reliability and restricts broader practical deployment. Among the hallucination mitigation methods, feature steering emerges as a promising approach that reduces erroneous outputs in LVLMs without increasing inference costs. However, current methods apply uniform feature steering across all layers. This heuristic strategy ignores inter-layer differences, potentially disrupting layers unrelated to hallucinations and ultimately leading to performance degradation on general tasks. In this paper, we propose a plug-and-play framework called Locate-Then-Sparsify for Feature Steering (LTS-FS), which controls the steering intensity according to the hallucination relevance of each layer. We first construct a synthetic dataset comprising token-level and sentence-level hallucination cases. Based on this dataset, we introduce an attribution method based on causal interventions to quantify the hallucination relevance of each layer. With the attribution scores across layers, we propose a layerwise strategy that converts these scores into feature steering intensities for individual layers, enabling more precise adjustments specifically on hallucination-relevant layers. Extensive experiments across multiple LVLMs and benchmarks demonstrate that our LTS-FS framework effectively mitigates hallucination while preserving strong performance.

Figures (8)

• Figure 1: Current methods (e.g., Nullu yang2025nullu) mitigate hallucinations by uniformly steering features across layers, which (a) alters feature distributions and (b) leads to degraded performance on general tasks like MMMU. In contrast, we propose a layerwise steering framework, LTS-FS, which mitigates hallucinations more effectively (e.g., on CHAIR) while minimally perturbing the feature distributions, thus preserving more generalization ability.
• Figure 2: Hallucination examples at token level and sentence level.
• Figure 3: Overview of our LTS-FS framework. First, we build a bi-granularity dataset with token level and sentence level hallucinations. Then, based on the dataset, hallucination-relevant layers are located through intervention-based attribution. Finally, a layerwise strategy is applied to control the feature steering intensity across layers according to the attribution scores.
• Figure 4: Results of MME evaluation.
• Figure 5: Demonstration of our framework for hallucination mitigation on two examples of LLaVA-Bench using LLaVA-v1.5-7B.