NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving

TL;DR

NuScenes-SpatialQA introduces the first ground-truth–based benchmark for evaluating spatial understanding and reasoning in vision-language models within autonomous driving. It builds two automated pipelines—3D scene graph construction and QA generation—on NuScenes data, yielding ~3.5M QA pairs across six camera views and ground-truth LiDAR spatial information. Experimental results show VLMs retain strong qualitative spatial understanding but struggle with quantitative tasks, and spatially enhanced models improve qualitative performance without clear gains in quantitative QA or direct geometric reasoning. The work provides a rigorous, self-contained framework for probing spatial reasoning in driving contexts and highlights substantial future challenges in achieving reliable, precise spatial reasoning in VLMs.

Abstract

Recent advancements in Vision-Language Models (VLMs) have demonstrated strong potential for autonomous driving tasks. However, their spatial understanding and reasoning-key capabilities for autonomous driving-still exhibit significant limitations. Notably, none of the existing benchmarks systematically evaluate VLMs' spatial reasoning capabilities in driving scenarios. To fill this gap, we propose NuScenes-SpatialQA, the first large-scale ground-truth-based Question-Answer (QA) benchmark specifically designed to evaluate the spatial understanding and reasoning capabilities of VLMs in autonomous driving. Built upon the NuScenes dataset, the benchmark is constructed through an automated 3D scene graph generation pipeline and a QA generation pipeline. The benchmark systematically evaluates VLMs' performance in both spatial understanding and reasoning across multiple dimensions. Using this benchmark, we conduct extensive experiments on diverse VLMs, including both general and spatial-enhanced models, providing the first comprehensive evaluation of their spatial capabilities in autonomous driving. Surprisingly, the experimental results show that the spatial-enhanced VLM outperforms in qualitative QA but does not demonstrate competitiveness in quantitative QA. In general, VLMs still face considerable challenges in spatial understanding and reasoning.

Figures (3)

• Figure 1: Comprehensive experiments on our NuScenes-SpatialQA benchmark have demonstrated VLMs' performance on spatial understanding and reasoning abilities, including spatial relationship tasks (top) and quantitative spatial measurement tasks (bottom).§
• Figure 2: Overall framework of NuScenes-SpatialQA. The framework consists of two automated pipelines: (1) Scene Graph Generation, where a 3D scene graph is constructed using 3D annotations and instance-level captions generated from the auto-captioning process, and (2) QA Generation, where the constructed 3D scene graph is utilized to generate spatial question-answer pairs based on QA templates.
• Figure 3: Example QA pairs from NuScenes-SpatialQA benchmark.