AutoSpatial: Visual-Language Reasoning for Social Robot Navigation through Efficient Spatial Reasoning Learning

TL;DR

AutoSpatial tackles the lack of spatial grounding in Vision-Language Models for social robot navigation by introducing structured spatial grounding and a data-labeling pipeline that blends auto-labeled VQA with limited manual annotations. It employs a two-round VQA training scheme to build from per-pedestrian perception and prediction to joint scene-level reasoning, action, and explanations. Evaluations on the CODA and SNEI benchmarks show that the combination of auto-labeled data and targeted manual annotations yields substantial gains across perception & prediction, reasoning, action, and explanations, with robustness to data mixing ratios. The work demonstrates practical gains for socially compliant robot navigation and identifies future directions in richer human-behavior understanding and temporal dynamics.

Abstract

We present a novel method, AutoSpatial, an efficient approach with structured spatial grounding to enhance VLMs' spatial reasoning. By combining minimal manual supervision with large-scale Visual Question-Answering (VQA) pairs auto-labeling, our approach tackles the challenge of VLMs' limited spatial understanding in social navigation tasks. By applying a hierarchical two-round VQA strategy during training, AutoSpatial achieves both global and detailed understanding of scenarios, demonstrating more accurate spatial perception, movement prediction, Chain of Thought (CoT) reasoning, final action, and explanation compared to other SOTA approaches. These five components are essential for comprehensive social navigation reasoning. Our approach was evaluated using both expert systems (GPT-4o, Gemini 2.0 Flash, and Claude 3.5 Sonnet) that provided cross-validation scores and human evaluators who assigned relative rankings to compare model performances across four key aspects. Augmented by the enhanced spatial reasoning capabilities, AutoSpatial demonstrates substantial improvements by averaged cross-validation score from expert systems in: perception & prediction (up to 10.71%), reasoning (up to 16.26%), action (up to 20.50%), and explanation (up to 18.73%) compared to baseline models trained only on manually annotated data.

Figures (3)

• Figure 1: Overview of the AutoSpatial approach. The approach consists of two primary components: Structured Spatial Grounding (purple), which standardizes positional (e.g., 'Slightly to the right' and 'At a very close distance' for the pedestrian in the bounding box) and directional descriptions (e.g., 'Moving towards west'), and Data Labeling (blue), which combines auto-labeled Visual Question-Answer (VQA) pairs that focus on percetion, prediction, and CoT reasoning, with manual annotations that focus on percetion, prediction, CoT reasoning, final action, and explanation, and further formats them into a two-round VQA structure (see details in Fig. \ref{['fig:example_vqa']}).
• Figure 2: An example of the two-round VQA structure, where training data of each round follows the same format of $<$image, question, ground-truth answer$>$. Round 1 is auto-labeled, focusing on perception, prediction, and CoT reasoning for individual pedestrians, while Round 2 is optional and only available when the image input is manually annotated. It refines individual descriptions into a comprehensive scene-level understanding, incorporating group dynamics and higher-level spatial reasoning. The structured spatial grounding, though not explicitly shown here, is incorporated into the system prompt as part of the model input. Note that bounding boxes and color-pedestrian correspondence are only provided to the model during training.
• Figure 3: While LLaVA-M suffers from faulty spatial reasoning, leading to ambiguousor ineffective navigation decisions, AutoSpatial exhibits improved pedestrian identification and reasoning, when augmented with auto-labeled VQA pairs.