Failures in Perspective-taking of Multimodal AI Systems

TL;DR

This study applies techniques from cognitive and developmental science to assess the perspective-taking abilities of GPT-4o and enables a comparison between the cognitive development of the human brain and that of multimodal AI.

Abstract

This study extends previous research on spatial representations in multimodal AI systems. Although current models demonstrate a rich understanding of spatial information from images, this information is rooted in propositional representations, which differ from the analog representations employed in human and animal spatial cognition. To further explore these limitations, we apply techniques from cognitive and developmental science to assess the perspective-taking abilities of GPT-4o. Our analysis enables a comparison between the cognitive development of the human brain and that of multimodal AI, offering guidance for future research and model development.

Figures (4)

• Figure 1: Examples of stimuli. Level 2 visual-judgment stimuli were cropped to improve OCR.
• Figure 2: GPT-4o performance on Level 1 stimuli.
• Figure 3: GPT-4o performance on Level 2 tasks compared to human response times.
• Figure 4: GPT-4o performance on Level 2 spatial-judgment stimuli with chain-of-thought prompting.