AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation Editing

TL;DR

Object hallucination in LVLMs stems from language bias, causing category, attribute, and relation errors. AFTER tackles this by coupling FAS, which textualizes ground-truth facts into category/attribute/relation descriptions, with QAO, which provides per-query offsets for adaptive activation editing. Experiments across POPE, MME, and AMBER demonstrate strong hallucination mitigation across three LVLMs while preserving general visual-language capabilities and maintaining fast inference. The approach offers a practical, generalizable path to trustworthy LVLMs with minimal overhead and data requirements.

Abstract

Large Vision-Language Models (LVLMs) have achieved substantial progress in cross-modal tasks. However, due to language bias, LVLMs are susceptible to object hallucination, which can be primarily divided into category, attribute, and relation hallucination, significantly impeding the trustworthy AI applications. Editing the internal activations of LVLMs has shown promising effectiveness in mitigating hallucinations with minimal cost. However, previous editing approaches neglect the effective guidance offered by factual textual semantics, thereby struggling to explicitly mitigate language bias. To address these issues, we propose Adaptive Factual-guided Visual-Textual Editing for hallucination mitigation (AFTER), which comprises Factual-Augmented Activation Steering (FAS) and Query-Adaptive Offset Optimization (QAO), to adaptively guides the original biased activations towards factual semantics. Specifically, FAS is proposed to provide factual and general guidance for activation editing, thereby explicitly modeling the precise visual-textual associations. Subsequently, QAO introduces a query-aware offset estimator to establish query-specific editing from the general steering vector, enhancing the diversity and granularity of editing. Extensive experiments on standard hallucination benchmarks across three widely adopted LVLMs validate the efficacy of the proposed AFTER, notably achieving up to a 16.3% reduction of hallucination over baseline on the AMBER benchmark. Our code and data will be released for reproducibility.

Figures (5)

• Figure 1: The above figure demonstrates the three types of hallucinations (category, attribute, and relation) caused by language bias. The below figure shows the comparisons between previous activation editing methods and AFTER.
• Figure 2: An overview of the AFTER. FAS first establishes the general and positive visual-textual editing direction with the guidance of facts. QAO then achieves precise query-adaptive editing by training a query-aware offset estimator, thereby explicitly mitigating language bias.
• Figure 3: Comparison of AFTER with SOTA editing methods on other perception and cognition capabilities on MME.
• Figure 4: Analysis on LLaVA-v1.5. Left: Ablation of number $K$ and strength $\alpha$. Right: Distribution of vector magnitudes.
• Figure 5: Deep analysis on LLaVA-v1.5. Left: Visualization of distinct activations yielded by the last layer. Right: Comparison of inference speed and hallucination mitigation.