Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models

TL;DR

Across all models, a significant performance drop when perspective-taking is required is observed, and performance in object detection tasks is poorly correlated with performance on VPT tasks, suggesting that the existing benchmarks might not be sufficient to understand this problem.

Abstract

Visual perspective-taking (VPT), the ability to understand the viewpoint of another person, enables individuals to anticipate the actions of other people. For instance, a driver can avoid accidents by assessing what pedestrians see. Humans typically develop this skill in early childhood, but it remains unclear whether the recently emerging Vision Language Models (VLMs) possess such capability. Furthermore, as these models are increasingly deployed in the real world, understanding how they perform nuanced tasks like VPT becomes essential. In this paper, we introduce two manually curated datasets, Isle-Bricks and Isle-Dots for testing VPT skills, and we use it to evaluate 12 commonly used VLMs. Across all models, we observe a significant performance drop when perspective-taking is required. Additionally, we find performance in object detection tasks is poorly correlated with performance on VPT tasks, suggesting that the existing benchmarks might not be sufficient to understand this problem. The code and the dataset will be available at https://sites.google.com/view/perspective-taking

Figures (7)

• Figure 1: Examples from Isle-Bricks and Isle-Dots including control questions checking general object detection ability and questions about Visual Perspective Taking. We open-source the datasets and the evaluation protocol.
• Figure 2: Our study shows that VLMs achieve poor performance in VPT tasks. Compared to the control task that does not require perspective-taking, the models suffer on average $32\%$ and $38\%$ drop in performance on Isle-Bricks and Isle-Dots respectively. The performance on the VPT task is often close to random chance.
• Figure 3: We report results for VPT tasks from Isle-Bricks and Isle-Dots given 0-shot and CoT prompting.
• Figure 4: Some models exhibit particularly low consistency scores, indicating proneness for positional bias.
• Figure 5: We report the percentage of model answers that we classified as Unknown (i.e., neither A nor B). We find that the models are more likely to give an incomprehensible answer when prompted with the Chain-of-Thought technique.