Investigating VLM Hallucination from a Cognitive Psychology Perspective: A First Step Toward Interpretation with Intriguing Observations

TL;DR

This work reframes hallucinations in Vision-Language Models through a cognitive psychology lens by introducing the AIpsych benchmark to diagnose biases such as authority bias and sycophancy, alongside logical inconsistency. It systematically analyzes how model size and architecture influence these biases and validates findings with a human subject study, revealing parallels and gaps between human and machine behavior. AIpsych combines 3,000 images and 60,000 questions to move beyond simply measuring hallucination frequency toward understanding why it occurs, enabling more interpretable and trustworthy multimodal systems. The findings highlight the need for cognitive-bias-aware evaluation and targeted debiasing strategies that address underlying reasoning patterns rather than surface-level output adjustments.

Abstract

Hallucination is a long-standing problem that has been actively investigated in Vision-Language Models (VLMs). Existing research commonly attributes hallucinations to technical limitations or sycophancy bias, where the latter means the models tend to generate incorrect answers to align with user expectations. However, these explanations primarily focus on technical or externally driven factors, and may have neglected the possibility that hallucination behaviours might mirror cognitive biases observed in human psychology. In this work, we introduce a psychological taxonomy, categorizing VLMs' cognitive biases that lead to hallucinations, including sycophancy, logical inconsistency, and a newly identified VLMs behaviour: appeal to authority. To systematically analyze these behaviours, we design AIpsych, a scalable benchmark that reveals psychological tendencies in model response patterns. Leveraging this benchmark, we investigate how variations in model architecture and parameter size influence model behaviour when responding to strategically manipulated questions. Our experiments reveal that as model size increases, VLMs exhibit stronger sycophantic tendencies but reduced authority bias, suggesting increasing competence but a potential erosion of response integrity. A human subject study further validates our hypotheses and highlights key behavioural differences between VLMs and human respondents. This work suggests a new perspective for understanding hallucination in VLMs and highlights the importance of integrating psychological principles into model evaluation.

Figures (8)

• Figure 1: An illustration of the cognitive biases in models. Left: a VLM exhibits sycophancy by favouring the questioner’s options despite recognising it is a pink cup. Right: a human demonstrates authority bias by accepting the question’s framing, also yielding the wrong answer. However, to distinguish between them, we will need to ask more questions. See Figure \ref{['fig:question']}.
• Figure 2: A sample image (top) with its question set from AIpsych, and an illustration of the classification flow (bottom). If a model selects the trap option for the first prompt, subsequent prompts are presented to probe its psychological behaviour. Colored phrases highlight representative prompt elements.
• Figure 3: Plot of experimental results. In general, as model size increases, plots $A$ and $F$ suggest an increasing trend in sycophancy and $ReS$, plot $B$ suggests a declining authority bias, and other details are discussed in the paper. Outliers are shown with dotted lines for clarity.
• Figure 4: Plots of Type I Sycophancy Rate (left), Type II Sycophancy Rate (center), and Full Response Rate (right).
• Figure 5: A screenshot of the human survey form.