RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

TL;DR

RoboSpatial addresses the deficit in spatial understanding within vision-language models used for robotics by introducing a large-scale, 2D/3D-ready dataset that encodes spatial relationships across multiple reference frames. The authors present a two-stage data generation pipeline that yields about 3 million QA pairs from 1 million images and 5k 3D scans, along with RoboSpatial-Val for evaluation and RoboSpatial-Home for real-world grounding. Experiments show that fine-tuning 2D and 3D VLMs on RoboSpatial improves spatial reasoning, generalization to unseen relationships, and performance in robot manipulation tasks, with real robot experiments validating practical impact. The work also provides an auxiliary grounding dataset to improve object reference resolution and discusses cross-domain generalization across indoor and tabletop scenes. Overall, RoboSpatial offers a scalable resource to teach spatial understanding for robotics and a framework for evaluating cross-modal spatial reasoning in embodied settings.

Abstract

Spatial understanding is a crucial capability that enables robots to perceive their surroundings, reason about their environment, and interact with it meaningfully. In modern robotics, these capabilities are increasingly provided by vision-language models. However, these models face significant challenges in spatial reasoning tasks, as their training data are based on general-purpose image datasets that often lack sophisticated spatial understanding. For example, datasets frequently do not capture reference frame comprehension, yet effective spatial reasoning requires understanding whether to reason from ego-, world-, or object-centric perspectives. To address this issue, we introduce RoboSpatial, a large-scale dataset for spatial understanding in robotics. It consists of real indoor and tabletop scenes, captured as 3D scans and egocentric images, and annotated with rich spatial information relevant to robotics. The dataset includes 1M images, 5k 3D scans, and 3M annotated spatial relationships, and the pairing of 2D egocentric images with 3D scans makes it both 2D- and 3D- ready. Our experiments show that models trained with RoboSpatial outperform baselines on downstream tasks such as spatial affordance prediction, spatial relationship prediction, and robot manipulation.

Figures (7)

• Figure 1: RoboSpatial dataset facilitates 3D spatial reasoning for robot manipulation. This illustration demonstrates how a model trained on RoboSpatial enables human-aligned spatial reasoning within the correct reference frame, supporting task grounding, planning, and detection for manipulation tasks.
• Figure 2: Overview of the RoboSpatial dataset. We automatically generate spatial relationship annotations from existing datasets with 3D point clouds, egocentric images, and 3D bounding box annotations. We create question/answer pairs covering three classes of spatial relationships, three spatial reference frames, and both binary (yes/no) and numeric (e.g.2D image points) answers. From 1M images and 5k scans, we generate over 3M spatial question/answer pairs.
• Figure 3: In-domain (RoboSpatial-Val, top) and out-of-domain (RoboSpatial-Home, BLINK fu2024blink, middle and bottom) results for RoboSpatial-trained models. Two models shown: SL (SpaceLLaVA chen2024spatialvlm) and RP (RoboPoint yuan2024robopoint); the -FT suffix indicates fine-tuning on RoboSpatial. Correct answers in green. All images except bottom-right in the out-of-domain rows are from RoboSpatial-Home.
• Figure 4: Robotics experiments: the red dot shows the model output (if not present, the model failed to provide a valid point in the image); green dots are used to show when a model outputs multiple points. The robot motion generator, cuRobo sundaralingam2023curobo, is used to grasp the item referenced by the generated point. The spatial- prefix indicates model trained with RoboSpatial.
• Figure 5: An example of generated top-down map of the image from 3D bounding boxes.