Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding

TL;DR

This work tackles LVLM hallucinations by introducing Tri-layer Contrastive Decoding (TCD), a training-free framework that leverages a watermark embedded in the input image to identify a visually grounded intermediate layer and then performs contrastive decoding across three layers: a mature top layer, an amateur layer, and the visually grounded pivot. The approach uses an adaptive constraint to merge logits from these layers, guiding generation toward factual alignment with the visual input and query. Empirical results on POPE, MME, and AMBER demonstrate state-of-the-art hallucination reduction and improved visual grounding, with ablations confirming the distinct contributions of the pivot layer and contrastive layers. The method is lightweight (no retraining) and scalable across LVLM backbones, offering a practical path toward more reliable multimodal models in real-world applications.

Abstract

Large Vision-Language Models (LVLMs) have recently shown promising results on various multimodal tasks, even achieving human-comparable performance in certain cases. Nevertheless, LVLMs remain prone to hallucinations -- they often rely heavily on a single modality or memorize training data without properly grounding their outputs. To address this, we propose a training-free, tri-layer contrastive decoding with watermarking, which proceeds in three steps: (1) select a mature layer and an amateur layer among the decoding layers, (2) identify a pivot layer using a watermark-related question to assess whether the layer is visually well-grounded, and (3) apply tri-layer contrastive decoding to generate the final output. Experiments on public benchmarks such as POPE, MME and AMBER demonstrate that our method achieves state-of-the-art performance in reducing hallucinations in LVLMs and generates more visually grounded responses.

Figures (8)

• Figure 1: Architectural comparison between (a) the conventional decoding method of LVLMs and (b) our proposed watermark-based tri-layer contrastive decoding method. To mitigate hallucinations in LVLMs, we leverage watermark for selecting visually grounded layer.
• Figure 2: An overview of TCD, which leverages a tri-layer contrastive decoding approach by dynamically selecting and comparing following three decoding layers: (i) mature layer, (ii) amateur layer, and (iii) visually well-grounded layer. The process involves embedding a watermark into the input image, posing an ad-hoc question (e.g., "What is the last captcha character in the image?"), and selecting the visually well-grounded layer. Note that the top layer is chosen as the mature layer, while the amateur layer is selected based on the highest JSD from the mature layer.
• Figure 3: Heatmaps of each LVLMs' layers selected with POPE datasets. (COCO, AOKVQA and GQA). As shown, the proposed methods showed consistent layer selection with low variance.
• Figure 4: Example generated by Base, VCD leng2023VCD, AVISC woo2024dontmissforesttrees and our method, given the question: "Describe this image." We randomly sample from Amber wang2023llm and all results are based on LLaVA-v1.5-7b model. We highlight phrases with red if they are not well visually grounded, and with blue if they are well visually grounded. We observe that our model successfully mitigates hallucinations compared to the other three baselines. Additional examples are provided in Appendix \ref{['fig: Qualitative_3']}.
• Figure 5: Comparison of accuracy across subsets of POPE-MSCOCO under varying $\lambda$ in the ablation setup, based on \ref{['eq: lambda-ablation']}. While the Random and Popular subsets show improved performance when the visual layer dominates (i.e., lower $\lambda$), the Adversarial subset benefits from a greater contributions of the amateur layer (i.e., larger $\lambda$), highlighting the distinct roles of the visual and amateur layers in mitigating different forms of hallucination.