MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments

TL;DR

The paper tackles socially compliant navigation by focusing on short-horizon, safety-critical decisions and proposes MUSON, a reasoning-oriented multimodal dataset with a five-step Chain-of-Thought annotation and a balanced six-action space. It provides a rigorous evaluation protocol across perception, reasoning, and action with a Human–AI–Human verification loop, and demonstrates that small Vision-Language Models can learn robust, socially aligned navigation when trained on MUSON. Qwen2.5-VL-3B achieves the best decision accuracy (0.8625) and strong semantic reasoning, validating MUSON as an effective benchmark for learning safe navigation from structured reasoning data. Overall, MUSON enables explicit reasoning supervision, addressing long-tail action issues and inconsistencies in prior datasets, and supports scalable learning for lightweight models in real-world urban environments.

Abstract

Socially compliant navigation requires structured reasoning over dynamic pedestrians and physical constraints to ensure safe and interpretable decisions. However, existing social navigation datasets often lack explicit reasoning supervision and exhibit highly long-tailed action distributions, limiting models' ability to learn safety-critical behaviors. To address these issues, we introduce MUSON, a multimodal dataset for short-horizon social navigation collected across diverse indoor and outdoor campus scenes. MUSON adopts a structured five-step Chain-of-Thought annotation consisting of perception, prediction, reasoning, action, and explanation, with explicit modeling of static physical constraints and a rationally balanced discrete action space. Compared to SNEI, MUSON provides consistent reasoning, action, and explanation. Benchmarking multiple state-of-the-art Small Vision Language Models on MUSON shows that Qwen2.5-VL-3B achieves the highest decision accuracy of 0.8625, demonstrating that MUSON serves as an effective and reusable benchmark for socially compliant navigation. The dataset is publicly available at https://huggingface.co/datasets/MARSLab/MUSON

Figures (5)

• Figure 1: The MUSON dataset construction pipeline. MUSON covers diverse real-world scenes across indoor and outdoor environments with varying crowd densities. Continuous navigation behaviors are discretized into six egocentric action categories with a rational distribution that preserves safety-critical corner cases to mitigate long-tail issues. Each sample is annotated using a structured five-step CoT, followed by a Human–AI–Human verification process to ensure annotation consistency and quality. MUSON supports multiple downstream tasks, including socially-compliant action planning, socially-aligned CoT reasoning, social-context scene perception, and socially-grounded navigation explanation.
• Figure 2: Visualization comparisons of samples from SNEI and MUSON.
• Figure 3: Training loss and gradient norm curves. (a) Fine-tuning loss curve of TinyLLaVA on MUSON. (b) Corresponding gradient norm curve.
• Figure 4: Visualization of high-level decision results for different SVLMs on MUSON.
• Figure 5: Visualization of high-level reasoning quality for different SVLMs on MUSON.