Self-Augmented Visual Contrastive Decoding

TL;DR

This work tackles hallucination in large vision-language models by introducing a training-free decoding strategy that couples query-aware visual augmentation with entropy-aware token selection. The method, SAVCD, uses SAS prompting to dynamically choose the most semantically disruptive augmentation based on the input query, and SAT to adapt the next-token candidate set according to logit entropy. Across four LVLM families and seven benchmarks, SAVCD delivers substantial improvements in factual consistency over state-of-the-art contrastive decoding approaches, while maintaining practical computation by enabling a single forward pass on text-only inputs in the lightweight mode. The framework highlights the value of aligning visual perturbations with textual intent and leveraging full logit distributions for robust, confidence-sensitive decoding in multimodal generation.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal capabilities, but they inherit the tendency to hallucinate from their underlying language models. While visual contrastive decoding has been proposed to mitigate this issue, existing methods often apply generic visual augmentations that disregard the specific context provided by the text query, limiting their effectiveness. This study introduces a novel training-free decoding strategy that addresses these limitations, featuring two key contributions. First, a self-augmentation prompting strategy that leverages the intrinsic knowledge of the model to dynamically align semantics between the query and the visual augmentation. Second, an adaptive thresholding algorithm that adaptively adjusts next token candidate size based on the output sparsity, utilizing full information from the logit distribution. Extensive experiments across four LVLMs and seven benchmarks demonstrate that the proposed decoding significantly enhances factual consistency compared to state-of-the-art decoding methods. This work highlights the importance of integrating query-dependent augmentation and entropy-aware decoding for improving effective generation of LVLMs.

Figures (10)

• Figure 1: Overview of the proposed augmentation choice process and sparsity adaptive truncation.
• Figure 2: Qualitative examples of SAVCD on MM-Vet yu2023mm and LLaVA-Bench liu2023visual, and corresponding logit distributions and SAT thresholds by timestamp.
• Figure 3: Distribution of self-augmentation choice across model size and benchmarks. Oracle indicates GPT-4o-mini decisions.
• Figure 4: Comparison of the normalized entropy and proposed inverse-entropy function by $\gamma$.
• Figure 5: Qualitative results on MMVP tong2024eyes.