EscherVerse: An Open World Benchmark and Dataset for Teleo-Spatial Intelligence with Physical-Dynamic and Intent-Driven Understanding

TL;DR

This work defines Teleo-Spatial Intelligence (TSI) as the integration of Physical-Dynamic and Intent-Driven Reasoning, and introduces EscherVerse, the first large-scale open-world benchmark and dataset for evaluating both pillars in real-world video scenarios. It presents a scalable data-curation pipeline (multi-stage filtering, LLM-based semantic scoring, red-teaming QA generation, and multi-model verification) and a dataset split into Escher-Bench (8k QA for evaluation) and Escher-35k (35k QA with Chain-of-Thought for training), all designed to probe object permanence, state transitions, trajectory prediction, and intent inference. Experimental results across 27 vision-language models reveal that even the strongest systems struggle with Teleo-Spatial tasks (top accuracy around 57%), demonstrating a substantial gap and validating EscherVerse as a challenging testbed. Finetuning on Escher-35k substantially improves performance, especially on Action & Intent-Driven reasoning, highlighting the dataset’s effectiveness in teaching intent-aware spatial understanding. The work also discusses limitations, societal implications, and directions for future research, including long-horizon planning and social spatial dynamics.

Abstract

The ability to reason about spatial dynamics is a cornerstone of intelligence, yet current research overlooks the human intent behind spatial changes. To address these limitations, we introduce Teleo-Spatial Intelligence (TSI), a new paradigm that unifies two critical pillars: Physical-Dynamic Reasoning--understanding the physical principles of object interactions--and Intent-Driven Reasoning--inferring the human goals behind these actions. To catalyze research in TSI, we present EscherVerse, consisting of a large-scale, open-world benchmark (Escher-Bench), a dataset (Escher-35k), and models (Escher series). Derived from real-world videos, EscherVerse moves beyond constrained settings to explicitly evaluate an agent's ability to reason about object permanence, state transitions, and trajectory prediction in dynamic, human-centric scenarios. Crucially, it is the first benchmark to systematically assess Intent-Driven Reasoning, challenging models to connect physical events to their underlying human purposes. Our work, including a novel data curation pipeline, provides a foundational resource to advance spatial intelligence from passive scene description toward a holistic, purpose-driven understanding of the world.

Figures (8)

• Figure 1: The concept of Teleo-Spatial Intelligence (TSI) in contrast to current paradigms. Current approaches are fundamentally object-centric. They are limited to Physical-Dynamic Reasoning—understanding how objects move and interact—but fail to grasp the underlying human purpose behind these changes. Our proposed TSI is a human-centric paradigm that unifies physical dynamics with Intent-Driven Reasoning. This synergy enables a holistic comprehension by inferring why spatial changes occur from how they happen.
• Figure 2: Cases of the six dimensions of TSI: represents Human-centric, while represents Object-Centric. Object-Centric does not imply the absence of humans, it signifies that the problem focuses more on physical laws or mechanical processes.
• Figure 3: The EscherVerse construction pipeline consists of the filtering and generation pipeline and the verification pipeline.
• Figure 4: Data distribution of the Escher-Bench.
• Figure 5: A failure in Dynamic Spatial Relationships. The failed models process a simple egocentric path, failing to build a unified scene representation.