SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models

TL;DR

SpaRRTa introduces a photorealistic synthetic benchmark to evaluate abstract spatial awareness in Visual Foundation Models by predicting relative object positions from images under egocentric and allocentric viewpoints. Using Unreal Engine 5, SpaRRTa provides controlled, ground-truth spatial labels across five diverse environments and probes frozen VFMs with linear, AbMILP, and efficient heads to reveal where spatial information is stored. Results show that spatial reasoning largely lives in patch-level representations and is easily masked by global pooling; 3D supervision enhances patch-level geometric structure and improves performance on perspective-taking tasks, especially with efficient probing. The study demonstrates that SpaRRTa captures a distinct spatial capability different from semantic classification and depth/pose tasks, offering a diagnostic tool to guide the development of spatially aware visual models for embodied applications.

Abstract

Visual Foundation Models (VFMs), such as DINO and CLIP, excel in semantic understanding of images but exhibit limited spatial reasoning capabilities, which limits their applicability to embodied systems. As a result, recent work incorporates some 3D tasks (such as depth estimation) into VFM training. However, VFM performance remains inconsistent across other spatial tasks, raising the question of whether these models truly have spatial awareness or overfit to specific 3D objectives. To address this question, we introduce the Spatial Relation Recognition Task (SpaRRTa) benchmark, which evaluates the ability of VFMs to identify relative positions of objects in the image. Unlike traditional 3D objectives that focus on precise metric prediction (e.g., surface normal estimation), SpaRRTa probes a fundamental capability underpinning more advanced forms of human-like spatial understanding. SpaRRTa generates an arbitrary number of photorealistic images with diverse scenes and fully controllable object arrangements, along with freely accessible spatial annotations. Evaluating a range of state-of-the-art VFMs, we reveal significant disparities between their spatial reasoning abilities. Through our analysis, we provide insights into the mechanisms that support or hinder spatial awareness in modern VFMs. We hope that SpaRRTa will serve as a useful tool for guiding the development of future spatially aware visual models.

Figures (19)

• Figure 1: Effective spatial reasoning requires awareness of spatial relations between visible objects. To analyze such spatial awareness of Visual Foundation Models (VFMs), we introduce Spatial Relation Recognition Task (SpaRRTa). We generate images showing different spatial layouts of several objects, encode them with VFMs, and probe whether their latent representations reliably encode the information about the spatial relations between the objects from a selected perspective.
• Figure 2: SpaRRTa evaluates the efficacy of encoding the spatial relations between objects in the image by VFM, either from the camera (Egocentric) or an arbitrary object's (Allocentric) perspective (see the right-most image). We use Unreal Engine 5 to produce photorealistic test images that are in-distribution for contemporary VFMs, using a variety of scene environments and object assets.
• Figure 3: Evaluation pipeline: The SpaRRTa evaluation setup consists of a data generation system built on Unreal Engine 5. The pipeline begins with an evaluation controller which sets the scene in a precisely controlled manner using a diverse set of realistic assets (1), and places the camera (2) for capture. Next, a near-photorealistic image is rendered by Unreal (3), and ground truth positional information is acquired (4). The image is then passed trough evaluated VFM, and resulting representation is fed to the probing head for the positional relation prediction (5). Finally, the accuracy is computed (6).
• Figure 4: Visualization of the Spatial Relation Recognition Task (SpaRRTa). The task is to predict the spatial relationship between the source (i.e. car) and target (tree) objects with respect to a given viewpoint (i.e. the camera in the egocentric variant (a), and human in the allocentric variant (b)) To eliminate ambiguity of the tasks, we generate scenes where the spatial relationships of objects are clearly recognizable -- the green area denotes valid locations for target placement.
• Figure 5: SpaRRTa Asset Library. SpaRRTa constructs test examples using a curated set of diverse, high-fidelity 3D assets selected based on common ImageNet classes deng2009imagenet.