Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models

Abstract

Multimodal language models (MLMs) perform well on semantic vision-language tasks but fail at spatial reasoning that requires adopting another agent's visual perspective. These errors reflect a persistent egocentric bias and raise questions about whether current models support allocentric reasoning. Inspired by human spatial cognition, we introduce perspective tokens, specialized embeddings that encode orientation through either (1) embodied body-keypoint cues or (2) abstract representations supporting mental rotation. Integrating these tokens into LLaVA-1.5-13B yields performance on level-2 visual perspective-taking tasks. Across synthetic and naturalistic benchmarks (Isle Bricks V2, COCO, 3DSRBench), perspective tokens improve accuracy, with rotation-based tokens generalizing to non-human reference agents. Representational analyses reveal that fine-tuning enhances latent orientation sensitivity already present in the base model, suggesting that MLMs contain precursors of allocentric reasoning but lack appropriate internal structure. Overall, embedding cognitively grounded spatial structure directly into token space provides a lightweight, model-agnostic mechanism for perspective-taking and more human-like spatial reasoning.

Figures (4)

• Figure 1: Overview of evaluation and training data. a, Stimuli from the perspective-taking benchmark systematically vary the reference avatar's orientation across degrees of alignment (0°–360°). b, Token construction methods: embodiment tokens encode body keypoint coordinates and calculated yaw orientation (above); rotation tokens encode object bounding box coordinates and azimuth labels from OrientAnything (below). c, Curriculum learning structure showing progression from atomic token generation to chain-of-thought reasoning to direct answer formats. The training schedule increases the proportion of chain-of-thought and direct items by 10% each epoch. d, Representative items from each evaluation benchmark (Perspective-Taking, Isle Bricks V2, COCO validation, 3DSRBench).
• Figure 2: LLaVA with embodiment tokens succeeds at unaligned perspective-taking tasks. a, LLaVA 1.5 13B fails at unaligned angles on the perspective-taking benchmark (left) while LLaVA with embodiment tokens succeeds (right). b, Performance across benchmarks: LLaVA with embodiment tokens substantially outperforms the base model on unaligned items from the perspective-taking benchmark, Isle Bricks V2, and COCO validation sets. The text-based control (embodiment information presented in natural language) improves over baseline but falls short of token-based performance. c-d, Response comparisons for aligned (c) and unaligned (d) queries showing base model (direct prompt), text-based model (chain-of-thought), and token-based model (chain-of-thought).
• Figure 3: Intermediate activations show alignment sensitivity in both embodiment token and base LLaVA. a, Number of feature selective units in base LLaVA 1.5 13B versus embodiment token model, measured at the multimodal projector layer (left) and language model layer (right). b-c, Average activation profiles of alignment-selective units as a function of reference angle, for units preferring unaligned over aligned conditions (b) and units preferring aligned over unaligned conditions (c). The approximately mirror-symmetric activation patterns reflect the use of standardized (z-scored) unit activations, such that preference for one condition implies reduced activation for the opposite condition.
• Figure 4: Similar to embodiment tokens, LLaVA with rotation tokens succeeded at unaligned perspective-taking tasks. a, Unlike LLaVA 1.5 13B (left), LLaVA with rotation tokens succeeded on unaligned items on the perspective-taking benchmark (right). b, Performance across benchmarks: LLaVA with rotation tokens substantially outperforms the base model on unaligned items from the perspective-taking benchmark, Isle Bricks V2, and COCO validation sets. The text-based control (embodiment information in natural language) improves over baseline but falls short of token-based performance. c-d, Response comparisons for aligned (c) and unaligned (d) queries showing base model (direct prompt), text-based model (chain-of-thought), and token-based model (chain-of-thought). e, Examples of LLaVA with rotation tokens generalizing to non-human references like furniture and animals from 3DSRBench.