Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study

Guanlin Wu, Boyan Su, Yang Zhao, Pu Wang, Yichen Lin, Hao Frank Yang

TL;DR

this work addresses the problem of enabling physics-informed spatial reasoning in foundation models for autonomous driving. it introduces the Spatial Intelligence Grid (sig), a grid-based scene representation that encodes object layouts, inter-object relations, and physical priors, together with the sigbench benchmark and three evaluation metrics to quantify intrinsic vsi. through few-shot in-context learning with state-of-the-art multimodal llms, sig-based prompts yield larger, more stable gains across vsi metrics than traditional vqa prompts, demonstrating sig's potential as a data schema for learning spatial intelligence. the results show sigbench supports both grid-based machine vsi and human-like, attention-driven vsi, with cross-domain generalization indicating practical relevance for physics-aware visual reasoning in autonomous driving.

Abstract

How to integrate and verify spatial intelligence in foundation models remains an open challenge. Current practice often proxies Visual-Spatial Intelligence (VSI) with purely textual prompts and VQA-style scoring, which obscures geometry, invites linguistic shortcuts, and weakens attribution to genuinely spatial skills. We introduce Spatial Intelligence Grid (SIG): a structured, grid-based schema that explicitly encodes object layouts, inter-object relations, and physically grounded priors. As a complementary channel to text, SIG provides a faithful, compositional representation of scene structure for foundation-model reasoning. Building on SIG, we derive SIG-informed evaluation metrics that quantify a model's intrinsic VSI, which separates spatial capability from language priors. In few-shot in-context learning with state-of-the-art multimodal LLMs (e.g. GPT- and Gemini-family models), SIG yields consistently larger, more stable, and more comprehensive gains across all VSI metrics compared to VQA-only representations, indicating its promise as a data-labeling and training schema for learning VSI. We also release SIGBench, a benchmark of 1.4K driving frames annotated with ground-truth SIG labels and human gaze traces, supporting both grid-based machine VSI tasks and attention-driven, human-like VSI tasks in autonomous-driving scenarios.

Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study

TL;DR

this work addresses the problem of enabling physics-informed spatial reasoning in foundation models for autonomous driving. it introduces the Spatial Intelligence Grid (sig), a grid-based scene representation that encodes object layouts, inter-object relations, and physical priors, together with the sigbench benchmark and three evaluation metrics to quantify intrinsic vsi. through few-shot in-context learning with state-of-the-art multimodal llms, sig-based prompts yield larger, more stable gains across vsi metrics than traditional vqa prompts, demonstrating sig's potential as a data schema for learning spatial intelligence. the results show sigbench supports both grid-based machine vsi and human-like, attention-driven vsi, with cross-domain generalization indicating practical relevance for physics-aware visual reasoning in autonomous driving.

Abstract

How to integrate and verify spatial intelligence in foundation models remains an open challenge. Current practice often proxies Visual-Spatial Intelligence (VSI) with purely textual prompts and VQA-style scoring, which obscures geometry, invites linguistic shortcuts, and weakens attribution to genuinely spatial skills. We introduce Spatial Intelligence Grid (SIG): a structured, grid-based schema that explicitly encodes object layouts, inter-object relations, and physically grounded priors. As a complementary channel to text, SIG provides a faithful, compositional representation of scene structure for foundation-model reasoning. Building on SIG, we derive SIG-informed evaluation metrics that quantify a model's intrinsic VSI, which separates spatial capability from language priors. In few-shot in-context learning with state-of-the-art multimodal LLMs (e.g. GPT- and Gemini-family models), SIG yields consistently larger, more stable, and more comprehensive gains across all VSI metrics compared to VQA-only representations, indicating its promise as a data-labeling and training schema for learning VSI. We also release SIGBench, a benchmark of 1.4K driving frames annotated with ground-truth SIG labels and human gaze traces, supporting both grid-based machine VSI tasks and attention-driven, human-like VSI tasks in autonomous-driving scenarios.
Paper Structure (28 sections, 13 equations, 11 figures, 6 tables)

This paper contains 28 sections, 13 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Grid-based Visual-Spatial Intelligence
  5. SIG-based In-context Learning for VSI
  6. Human-Like VSI with Grid
  7. Experiment
  8. SIG Benchmark
  9. Model Selection
  10. Results and Analysis for Grid-based VSI on SIGBench
  11. Results and Analysis for Human-Like VSI with Grid on SIGBench
  12. Conclusion and Discussion
  13. Acknowledgement
  14. Evaluation Metrics
  15. Multi-level Spatial Matching
  16. ...and 13 more sections

Figures (11)

  • Figure 1: Examples of Human VSI in Painting. Abraham Bosse, a French artist and theorist illustrates a systematic, grid-based method for achieving visual-spatial correlations in rendering three-dimensional space on a two-dimensional canvas (left) bosse1algemeene, and he employed this grid-based visual scheme in portrait painting (middle) scott2013screenAlbertiVeil. Procedures of drawing a cast with graphical priors from scratch (right). chase1919drawing
  • Figure 2: Overview of Human-like SIG in AD Scenario. In the left, we use SIG to represent the spatial relation of traffic sign, traffic lights, vehicles and self (ego-vehicle) in the image. We apply homographic transformation to convert human gaze attention from image to SIG size in the right. Combining them, we get human-like SIG, which can then be extracted to human-like SRG and SRP in middle part. The order denotes the rank of an object of the same category in the image from left to right (e.g. black truck 1 is the left-most object among vehicles).
  • Figure 3: Illustration Examples of MLSM and SRGS. At the start of both MLSM and SRGS, it will match the objects between the predicted and GT SIG using bipartite matching. For MLSM, we provide example for calculating TP, FP and FN for vehicles in the boxed area in upper part. For SRGS, we highlight the node and edge needed for insertion and substitution and their total edit distance in lower part.
  • Figure 4: Directional Relation Circle. The semantic relational distance between any two prepositions is the smallest step count around the circle (e.g. between “at the back left of” and “at the front of” is 3 instead of 5).
  • Figure 5: (a) is the annotation pipeline of $\mathtt{SIGBench}$ and (b) illustrates the SIGC and SRPF tasks in $\mathtt{SIGBench}$.
  • ...and 6 more figures