Context-Aware Decoding for Faithful Vision-Language Generation

TL;DR

This work probes the layer-wise generation dynamics that drive hallucinations and proposes a training-free mitigation strategy, CEI, a lightweight method that harnesses the hidden state of the last input token-the context embedding-as a grounding signal to maintain visual fidelity throughout decoding and curb hallucinations.

Abstract

Hallucinations, generating responses inconsistent with the visual input, remain a critical limitation of large vision-language models (LVLMs), especially in open-ended tasks such as image captioning and visual reasoning. In this work, we probe the layer-wise generation dynamics that drive hallucinations and propose a training-free mitigation strategy. Employing the Logit Lens, we examine how LVLMs construct next-token distributions across decoder layers, uncovering a pronounced commitment-depth gap: truthful tokens accumulate probability mass on their final candidates earlier than hallucinatory ones. Drawing on this discovery, we introduce Context Embedding Injection (CEI), a lightweight method that harnesses the hidden state of the last input token-the context embedding-as a grounding signal to maintain visual fidelity throughout decoding and curb hallucinations. Evaluated on the CHAIR, AMBER, and MMHal-Bench benchmarks (with a maximum token length of 512), CEI outperforms state-of-the-art baselines across three LVLMs, with its dynamic variant yielding the lowest overall hallucination rates. By integrating novel mechanistic insights with a scalable intervention, this work advances the mitigation of hallucinations in LVLMs.

Figures (11)

• Figure 1: Context embedding semantic alignment. Box plot of centered cosine similarities $\cos(\tilde{\mathbf{c}}, \tilde{\mathbf{w}})$ between the centered context embedding $\tilde{\mathbf{c}}$ and centered target token embedding $\tilde{\mathbf{w}}$. Blue and orange boxes denote truthful and hallucinatory tokens respectively.
• Figure 2: Layer-wise commitment curves. Top-$K$ probability mass, $M_K(\ell)$, of truthful (blue) and hallucinatory (red) tokens across decoder layers, with shaded bands indicating one standard deviation: (a) InstructBLIP and (b) LLaVA-1.5.
• Figure 3: Average confidence in decision set. Histogram of mean top-$K$ probability mass, $\bar{M}_K$, of truthful (blue) and hallucinatory (red) tokens for (a) InstructBLIP and (b) LLaVA-1.5 ($K=40$).
• Figure 4: CEI overview. An initial forward pass over the image--prompt input extracts a fixed context embedding$\mathbf{c}$ from the final decoder layer at the last prompt position (top). During autoregressive decoding, this pre-computed signal is injected at a chosen decoder layer via a weighted average mechanism, guiding generation toward the original image-conditioned context and improving visual grounding (bottom). Blue, green, and brown squares denote the embeddings of image tokens, prompt tokens, and previously generated tokens, respectively.
• Figure 5: Image captioning comparison. Comparing generated captions via the baseline methods and our CEI on a sample image.