Mechanisms of Prompt-Induced Hallucination in Vision-Language Models

TL;DR

This work investigates prompt-induced hallucinations (PIH) in vision–language models using a controlled object-counting task. By introducing misaligned prompts, the authors show that PIH emerges more when true counts are high, and that a small set of attention heads can causally mediate and suppress PIH through head ablations performed without retraining. The study finds that PIH heads reside in early LM layers and that their ablation shifts attention toward the image and reduces prompt copying, with generalization to a color-prediction task. Across three diverse VLMs, ablations reduce prompt-driven copying by substantial margins (up to 95% in color tasks) and improve visual grounding, though mechanisms are model-specific. These results highlight a localized, language-model-driven source of PIH and suggest practical, data-free interventions to improve alignment in multimodal systems.

Abstract

Large vision-language models (VLMs) are highly capable, yet often hallucinate by favoring textual prompts over visual evidence. We study this failure mode in a controlled object-counting setting, where the prompt overstates the number of objects in the image (e.g., asking a model to describe four waterlilies when only three are present). At low object counts, models often correct the overestimation, but as the number of objects increases, they increasingly conform to the prompt regardless of the discrepancy. Through mechanistic analysis of three VLMs, we identify a small set of attention heads whose ablation substantially reduces prompt-induced hallucinations (PIH) by at least 40% without additional training. Across models, PIH-heads mediate prompt copying in model-specific ways. We characterize these differences and show that PIH ablation increases correction toward visual evidence. Our findings offer insights into the internal mechanisms driving prompt-induced hallucinations, revealing model-specific differences in how these behaviors are implemented.

Figures (10)

• Figure 1: Example of prompt-induced hallucination (PIH) in numerical reasoning. The model correctly answers the baseline counting question. However, it conforms to the prompt and hallucinates additional waterlilies when asked to describe a mismatched number of objects. As the mismatch between the prompt and visual evidence grows, the model increasingly favors the image and recovers the correct count.
• Figure 2: PIH rates across different numbers of ground-truth objects. Blue lines show the percentage of outputs matching the prompted number for different offsets, $k$. Orange lines show matches to the true count. As the object count increases, models stop correcting the prompt.
• Figure 3: Base probabilities on the object counting task. We plot the probability assigned to the true count $N$ and neighboring counts $N+k$ ($k \in \{1,\ldots,5\}$) given the prompt "How many [objects] are in the image?". As object counts increase, confidence in $N$ decreases and probability mass spreads to neighboring counts.
• Figure 4: Image attention mass at the layer with the largest intervention-induced change for each model. Namely, LLaVA-OneVision (layer 2, $\Delta$=$0.121$), Qwen-VL (layer 27, $\Delta$=$0.053$),and Janus-pro (layer 22, $\Delta$=$0.037$). Gray bars indicate the baseline attention mass for image tokens, while green bars show an increase after PIH-knockout.
• Figure 5: Impact of knockout on the probability of the correct count in digit (blue) and word form (orange). We measure the probabilities given the prompt "Describe the $N_{\text{digit}}$ [objects]". The knockout effects are shown in green (increase) and red (decrease).