SITE: towards Spatial Intelligence Thorough Evaluation

TL;DR

SITE presents a comprehensive spatial intelligence benchmark for large vision-language models by unifying tasks from 31 datasets and adding two novel view-taking and dynamic tasks, all reformulated as MC-VQA. It combines bottom-up data collection with a top-down cognitive-science taxonomy to cover figural, vista, and environmental scales, as well as intrinsic/extrinsic and static/dynamic factors. Evaluations across 9 models reveal a substantial gap to human performance, particularly in spatial orientation and multi-view reasoning, while exposing a strong link between spatial intelligence and embodied robotics performance ($r=0.902$). The work highlights the need for viewpoint- and dynamics-aware training in VLMs and provides a scalable, cross-domain platform for ongoing SI assessment with practical implications for embodied AI systems.

Abstract

Spatial intelligence (SI) represents a cognitive ability encompassing the visualization, manipulation, and reasoning about spatial relationships, underpinning disciplines from neuroscience to robotics. We introduce SITE, a benchmark dataset towards SI Thorough Evaluation in a standardized format of multi-choice visual question-answering, designed to assess large vision-language models' spatial intelligence across diverse visual modalities (single-image, multi-image, and video) and SI factors (figural to environmental scales, spatial visualization and orientation, intrinsic and extrinsic, static and dynamic). Our approach to curating the benchmark combines a bottom-up survey about 31 existing datasets and a top-down strategy drawing upon three classification systems in cognitive science, which prompt us to design two novel types of tasks about view-taking and dynamic scenes. Extensive experiments reveal that leading models fall behind human experts especially in spatial orientation, a fundamental SI factor. Moreover, we demonstrate a positive correlation between a model's spatial reasoning proficiency and its performance on an embodied AI task.

Figures (14)

• Figure 1: We introduce SITE, a comprehensive benchmark for evaluating large vision-language models' spatial intelligence (SI). Three SI classification systems drawn from cognitive science, corresponding to the three panels, drive the design of SITE.
• Figure 2: Data collection pipeline for the bottom-up part of our benchmark. We conduct a large-scale effort to select image and video-based benchmarks that may contain SI tasks before using the GPT-4o model to filter out irrelevant evaluation samples. Finally, we generate 6 coarse categories and perform stratified sampling to obtain an even distribution over all SI categories.
• Figure 3: Ego-Exo view association tasks. The goal of this task is to pick the correct exocentric view given the egocentric view of a visual scene or vice versa.
• Figure 4: Category distribution by cognitive classification systems. Left: distribution before balancing. Right: our final benchmark's distribution.
• Figure 5: Two samples for each spatial category. (Part I)