Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models

TL;DR

This work probes whether Vision Language Models can perform visual perspective taking (VPT) beyond surface-level scene understanding. Using a tightly controlled LEGO-based dataset of 144 tasks, each paired with seven diagnostic questions, the authors quantify models' abilities across scene understanding, spatial reasoning, and VPT. Results show strong scene understanding across models, but pronounced deficits in spatial reasoning and even more so in VPT, with notable directional biases (e.g., east) that persist under multiple interventions. The findings argue for explicit geometric representations and targeted training to enable robust VPT, with implications for safety-critical, collaborative robotics and human–AI interaction scenarios.

Abstract

We investigate the ability of Vision Language Models (VLMs) to perform visual perspective taking using a novel set of visual tasks inspired by established human tests. Our approach leverages carefully controlled scenes, in which a single humanoid minifigure is paired with a single object. By systematically varying spatial configurations - such as object position relative to the humanoid minifigure and the humanoid minifigure's orientation - and using both bird's-eye and surface-level views, we created 144 unique visual tasks. Each visual task is paired with a series of 7 diagnostic questions designed to assess three levels of visual cognition: scene understanding, spatial reasoning, and visual perspective taking. Our evaluation of several state-of-the-art models, including GPT-4-Turbo, GPT-4o, Llama-3.2-11B-Vision-Instruct, and variants of Claude Sonnet, reveals that while they excel in scene understanding, the performance declines significantly on spatial reasoning and further deteriorates on perspective-taking. Our analysis suggests a gap between surface-level object recognition and the deeper spatial and perspective reasoning required for complex visual tasks, pointing to the need for integrating explicit geometric representations and tailored training protocols in future VLM development.

Figures (12)

• Figure 1: Prediction correctness across three categories of growing difficulty: scene understanding, spatial reasoning, and visual perspective taking. Error bars represent $95\%$ confidence intervals (estimated using bootstrapping (10,000 iterations)). The random classifier is a baseline choosing an answer uniformly at random, see \ref{['app:random_baseline']}.
• Figure 2: Sixteen tasks involving a single humanoid minifigure--object pair. Tasks vary by the object's placement (left, right, front, back); the orientation of the humanoid minifigure (facing toward or away from the object); and camera angle (surface-level an bird's-eye views). All images had the same dimensions, but some are enlarged here for presentation purposes.
• Figure 3: Prediction correctnesses on diagnostic questions.
• Figure 4: Comparison of co-occurrence matrix for models (columns) across questions Q5 (top row) and Q7 (bottom row). For more details see \ref{['app:matrix']}.
• Figure 5: Experiments to investigate GPT-4-Turbo’s persistent east bias in spatial reasoning (Q5). Top-left: Zoomed-in images of humanoid minifigures, testing the impact of increased visual detail. Bottom-left: Images with explicit cardinal direction labels (N, S, E, W) added. Top-right: Images with only a humanoid minifigure (no secondary objects). Bottom-right: Human figures replacing humanoid minifigures. Additionally, we permuted the cardinal directions in the prompt, testing all 24 possible orders of north, south, east, and west.