GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation

TL;DR

VLMs struggle with spatial reasoning; GRAID addresses this by generating large-scale spatial VQA data using only 2D bounding-box geometry from detectors, avoiding 3D reconstruction and caption-based hallucinations. The framework combines Scene Understanding with SPARQ, enabling fast, predicate-driven question realization that yields over $8.5\times 10^{6}$ VQA pairs with $91.16\%$ human-validated accuracy across BDD, NuImages, and Waymo, far exceeding a $57.6\%$ baseline from prior work. Fine-tuning a LLAMA-based VLM on GRAID data achieves transferable spatial concepts, improving held-out and cross-dataset performance and boosting public benchmarks such as BLINK and A-OKVQA (e.g., +$32.5\%$ overall on VLMEvalKit benchmarks). This scalable, open-source approach lowers the bar for high-quality spatial reasoning data and enables broader evaluation and deployment in embodied and non-embodied domains.

Abstract

Vision Language Models (VLMs) achieve strong performance on many vision-language tasks but often struggle with spatial reasoning$\unicode{x2014}$a prerequisite for many applications. Empirically, we find that a dataset produced by a current training data generation pipeline has a 57.6% human validation rate. These rates stem from current limitations: single-image 3D reconstruction introduces cascading modeling errors and requires wide answer tolerances, while caption-based methods require hyper-detailed annotations and suffer from generative hallucinations. We present GRAID, built on the key insight that qualitative spatial relationships can be reliably determined from 2D geometric primitives alone. By operating exclusively on 2D bounding boxes from standard object detectors, GRAID avoids both 3D reconstruction errors and generative hallucinations, resulting in datasets that are of higher quality than existing tools that produce similar datasets as validated by human evaluations. We apply our framework to the BDD100k, NuImages, and Waymo datasets, generating over 8.5 million high-quality VQA pairs creating questions spanning spatial relations, counting, ranking, and size comparisons. We evaluate one of the datasets and find it achieves 91.16% human-validated accuracy$\unicode{x2014}$compared to 57.6% on a dataset generated by recent work. Critically, we demonstrate that when trained on GRAID data, models learn spatial reasoning concepts that generalize: models fine-tuned on 6 question types improve on over 10 held-out types, with accuracy gains of 47.5% on BDD and 37.9% on NuImages for Llama 3.2B 11B, and when trained on all questions types, achieve improvements on several existing benchmarks such as BLINK. The GRAID framework, datasets, and additional information can be found $\href{this https URL}{here}$.

Figures (9)

• Figure 1: An example from the community implementation of SpatialVLM showing typical errors in current methods, and an example from the OpenSpatialDataset from SpatialRGPT, and GRAID-BDD. In the leftmost panel, although there are grammatical errors, the larger issue is that the answer is incorrect by a few orders of magnitude. In the middle panel, it is difficult to assign semantically meaningful names to each region and to determine which region of interest is being asked about. On the right, GRAID poses a deceptively simple question that requires a few steps of reasoning.
• Figure 2: Hierarchical breakdown of 5.3M visual questions generated by GRAID using Berkeley Deep Drive as the source images. There are five cognitive categories: Spatial Relations (53.5%), Counting (26.7%), Ranking & Extremes (14.9%), Localization (2.6%), and Size & Aspect (1.3%). Question details including runtime, and predicate and apply methods can be found in the Appendix.
• Figure 3: We fine-tune Llama 3.2 11B on only 6 questions from GRAID-BDD (hatched upper-left corner). Evaluations demonstrate a greater understanding across more difficult spatial reasoning questions in the GRAID-BDD validation set, generalization to a fifth topic not seen in training (Size & Aspect), and in all 19 question types never seen from GRAID-NuImages.
• Figure 4: After supervised fine-tuning on the GRAID-BDD dataset, we can see improvements in the model's ability to answer questions on the held-out questions of GRAID-BDD, and a dataset with a different distribution of scenes, GRAID-NuImages.
• Figure 5: Evaluation of ablated models on GRAID-BDD and GRAID-NuImages datasets