Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions

TL;DR

This work addresses whether spatial intelligence in AI systems stems from visual perception or the underlying reasoning backbone. It introduces SiT-Bench, a large-scale textual benchmark that decouples spatial cognition from perception by using coordinate-aware, pixel-free descriptions across 17 subtasks and five dimensions. Through extensive evaluation of SOTA LLMs and VLMs, the study reveals that while models excel at localized semantic tasks, global consistency and perspective transformation remain challenging, and explicit reasoning (chain-of-thought) substantially boosts performance, signaling latent world-modeling in LLMs. The benchmark, data pipeline, and analysis collectively provide a principled resource to guide the development of spatially-grounded LLM backbones for future vision-language systems and embodied agents, with the authors releasing code and data to support open research.

Abstract

Recent advancements in Spatial Intelligence (SI) have predominantly relied on Vision-Language Models (VLMs), yet a critical question remains: does spatial understanding originate from visual encoders or the fundamental reasoning backbone? Inspired by this question, we introduce SiT-Bench, a novel benchmark designed to evaluate the SI performance of Large Language Models (LLMs) without pixel-level input, comprises over 3,800 expert-annotated items across five primary categories and 17 subtasks, ranging from egocentric navigation and perspective transformation to fine-grained robotic manipulation. By converting single/multi-view scenes into high-fidelity, coordinate-aware textual descriptions, we challenge LLMs to perform symbolic textual reasoning rather than visual pattern matching. Evaluation results of state-of-the-art (SOTA) LLMs reveals that while models achieve proficiency in localized semantic tasks, a significant "spatial gap" remains in global consistency. Notably, we find that explicit spatial reasoning significantly boosts performance, suggesting that LLMs possess latent world-modeling potential. Our proposed dataset SiT-Bench serves as a foundational resource to foster the development of spatially-grounded LLM backbones for future VLMs and embodied agents. Our code and benchmark will be released at https://github.com/binisalegend/SiT-Bench .

Figures (3)

• Figure 1: Tasks Demonstration of SiT-Bench. Several representative subtasks are selected for demonstration in each of task categories. Note: The images shown are for illustrative purposes only to aid understanding; the actual evaluation uses only textual input without any visual data. The questions and captions above are slightly simplified for clarity and conciseness.
• Figure 2: Benchmark curation pipeline. The pipeline consists of two parallel paths: Path A generates QA pairs from scratch by collecting diverse scene images (robotic manipulation, urban streets, indoor spaces, simulations), applying GPT-4o quality scoring to filter spatially complex samples, and guiding VLMs to produce spatial QA pairs. Path B adapts existing vision benchmarks by selecting tasks solvable via pure text (e.g., multi-view reasoning, orientation), captioning their images, and filtering out tasks requiring absolute metrics. Both paths undergo DeepSeek-R1 automated filtering to eliminate data leakage (e.g., direct counting) and caption-uninferrable questions, followed by expert review with R1-CoT rationales to finalize 3,800 high-quality samples.
• Figure 3: The simplified thought process examples of Gemini-3-Flash. Complete reasoning process in Appendix \ref{['app:reasoning_process']}