From Cognition to Precognition: A Future-Aware Framework for Social Navigation

TL;DR

The paper tackles socially aware robot navigation in dynamic human environments by forecasting future human trajectories and shaping behavior through a social penalty. It introduces Falcon, a future-aware RL framework with a Social Cognition Penalty and a Spatial-Temporal Precognition Module, trained with depth and GPS+Compass inputs. The authors present two realistic SocialNav benchmarks, Social-HM3D and Social-MP3D, and demonstrate state-of-the-art results with about 55% task success and strong personal-space compliance, along with solid zero-shot generalization. This work advances realistic evaluation and training for socially compliant navigation, providing datasets and code to accelerate future research.

Abstract

To navigate safely and efficiently in crowded spaces, robots should not only perceive the current state of the environment but also anticipate future human movements. In this paper, we propose a reinforcement learning architecture, namely Falcon, to tackle socially-aware navigation by explicitly predicting human trajectories and penalizing actions that block future human paths. To facilitate realistic evaluation, we introduce a novel SocialNav benchmark containing two new datasets, Social-HM3D and Social-MP3D. This benchmark offers large-scale photo-realistic indoor scenes populated with a reasonable amount of human agents based on scene area size, incorporating natural human movements and trajectory patterns. We conduct a detailed experimental analysis with the state-of-the-art learning-based method and two classic rule-based path-planning algorithms on the new benchmark. The results demonstrate the importance of future prediction and our method achieves the best task success rate of 55% while maintaining about 90% personal space compliance. We will release our code and datasets. Videos of demonstrations can be viewed at https://zeying-gong.github.io/projects/falcon/ .

Figures (5)

• Figure 1: We integrate trajectory prediction into the SocialNav task. In (a), the robot navigates toward a goal while predicting human trajectories (dashed lines) and avoiding them, following social etiquette. The robot uses depth input as shown in (b). (c) offers a top-down map for reference, which is not used by the robot.
• Figure 2: Falcon Overview: The main policy network (top-right) takes Depth and GPS+Compass data as input. Its behavior is guided by the Social Cognition Penalty, which encourages socially compliant navigation and generates the main loss. During training, the output of the network's state encoder, combined with auxiliary information from the Habitat simulator, is processed by the Spatial-Temporal Precognition Module (bottom-right). Three socially-aware auxiliary tasks are then performed, producing auxiliary losses. The total loss is computed by weighting the main loss with the auxiliary losses.
• Figure 3: Human Distribution by Scene Area in Social-HM3D and Social-MP3D (Train/Test): Our benchmark balances social density for human-robot interactions while avoiding overcrowding.
• Figure 4: Comparisons of SocialNav Algorithms in Different Encounters: Our method outperforms other algorithms across various encounters. Green indicates safe behaviors, orange indicates risky behaviors (e.g., proximity to humans or collisions with obstacles), and red indicates unsafe behaviors (i.e., collisions with humans).
• Figure 5: Training Curve of SPL for Ablation Study: The full Falcon model with SPM and SCP converges faster and performs better.