Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception

TL;DR

This work tackles LVLM hallucinations by addressing two challenging dimensions: counterfactual presupposition questions (CPQs) and object-perception errors. It introduces Antidote, a unified synthetic-data post-training framework that decouples co-occurrence biases and injects factual priors to enable self-correction through Direct Preference Optimization, and CP-Bench, a two-part benchmark (dev and test) for evaluating CPQ discrimination and factual response generation. Across LLaVA-based models, Antidote delivers substantial improvements on CP-Bench and other hallucination benchmarks (POPE, CHAIR, SHR) without relying on external supervision or incurring significant catastrophic forgetting. The approach demonstrates that synthetic data pipelines and preference alignment can robustly reduce hallucinations while preserving general capabilities, offering practical benefits for deploying LVLMs in sensitive, cross-modal tasks.

Abstract

Large Vision-Language Models (LVLMs) have achieved impressive results across various cross-modal tasks. However, hallucinations, i.e., the models generating counterfactual responses, remain a challenge. Though recent studies have attempted to alleviate object perception hallucinations, they focus on the models' response generation, and overlooking the task question itself. This paper discusses the vulnerability of LVLMs in solving counterfactual presupposition questions (CPQs), where the models are prone to accept the presuppositions of counterfactual objects and produce severe hallucinatory responses. To this end, we introduce "Antidote", a unified, synthetic data-driven post-training framework for mitigating both types of hallucination above. It leverages synthetic data to incorporate factual priors into questions to achieve self-correction, and decouple the mitigation process into a preference optimization problem. Furthermore, we construct "CP-Bench", a novel benchmark to evaluate LVLMs' ability to correctly handle CPQs and produce factual responses. Applied to the LLaVA series, Antidote can simultaneously enhance performance on CP-Bench by over 50%, POPE by 1.8-3.3%, and CHAIR & SHR by 30-50%, all without relying on external supervision from stronger LVLMs or human feedback and introducing noticeable catastrophic forgetting issues.

Figures (6)

• Figure 1: The hallucination responses induced by CPQ. Though recent hallucination mitigation methods improves LVLMs in object perception, while their models can still be easily deceived by CPQs and induce severe hallucinatory responses.
• Figure 2: Performance comparison of LVLMs on benchmarks of general capabilities (MMBench liu2025mmbench), hallucination of object perception (POPE li2023evaluating) and CPQ (the proposed CP-Bench). Higher values indicate better performance on corresponding benchmarks.
• Figure 3: Examples of hallucination induced by CPQs and the synthetic samples of Antidote. The CPQs are selected from the test set of the proposed CP-Bench, which will be introduced in Section \ref{['sect_cp_bench']}. “Hallucination candidates” are the non-existent objects that commonly appear in similar scenes. More examples can be viewed in Appendix.
• Figure 4: The data synthesis pipeline for Antidote. The pipeline consists of three stages: (a) construction of caption Pool; (b) visual scene understanding; (c) data synthesis.
• Figure 5: Overview of the proposed Antidote post-training. The factual information from the synthetic data is seamlessly integrated into the input task prompt. The LVLMs can utilize this information to self-correct the responses as “positive” samples. For the original responses, they are regarded as “negative” samples to achieve preference alignment for hallucination alleviation.