PeRoI: A Pedestrian-Robot Interaction Dataset for Learning Avoidance, Neutrality, and Attraction Behaviors in Social Navigation

TL;DR

This work tackles the lack of datasets that capture diverse pedestrian responses to robots, proposing PeRoI, a large-scale, real-world trajectory dataset with explicit avoidance, neutrality, and attraction labels across two outdoor environments and three robot conditions. It introduces NeuRoSFM, a neural-augmented Social Force Model that learns five force components—including robot- and group-induced forces—via dedicated networks and combines them to predict pedestrian motion near robots. Empirical results show PeRoI enhances trajectory prediction compared to existing datasets and baselines, while NeuRoSFM achieves state-of-the-art accuracy across ETH, JRDB, and PeRoI with ablations confirming the value of robot and group forces. Overall, the dataset and modeling framework enable more realistic, socially aware navigation in human-centered spaces, accommodating the spectrum of pedestrian responses to robotic agents.

Abstract

Robots are increasingly being deployed in public spaces such as shopping malls, sidewalks, and hospitals, where safe and socially aware navigation depends on anticipating how pedestrians respond to their presence. However, existing datasets rarely capture the full spectrum of robot-induced reactions, e.g., avoidance, neutrality, attraction, which limits progress in modeling these interactions. In this paper, we present the Pedestrian-Robot Interaction~(PeRoI) dataset that captures pedestrian motions categorized into attraction, neutrality, and repulsion across two outdoor sites under three controlled conditions: no robot present, with stationary robot, and with moving robot. This design explicitly reveals how pedestrian behavior varies across robot contexts, and we provide qualitative and quantitative comparisons to established state-of-the-art datasets. Building on these data, we propose the Neural Robot Social Force Model~(NeuRoSFM), an extension of the Social Force Model that integrates neural networks to augment inter-human dynamics with learned components and explicit robot-induced forces to better predict pedestrian motion in vicinity of robots. We evaluate NeuRoSFM by generating trajectories on multiple real-world datasets. The results demonstrate improved modeling of pedestrian-robot interactions, leading to better prediction accuracy, and highlight the value of our dataset and method for advancing socially aware navigation strategies in human-centered environments.

Figures (6)

• Figure 1: Example scenario of a robot influencing the trajectories of nearby pedestrians, leading them to show one of three distinct behaviors: repulsion (red), neutral (blue), or attraction (green). The trajectories and the location present in the image are a visual representation of the ones in our PeRoI dataset.
• Figure 2: Distinct pedestrian behaviors when close to robots taken from our PeRoI dataset: (a) The pedestrian clearly avoids the static robot (star) while walking toward their goal. (b) The pedestrian walks close to the robot without any noticeable change in trajectory direction. (c) The pedestrian deviates from their original path to approach the robot before resuming their goal-directed movement.
• Figure 3: Outdoor environments used for data collection. (a) A pathway crossing with two office building entries. (b) A larger university campus open space.
• Figure 4: Robots used during the collection of the PeRoI dataset.
• Figure 5: Structure of our proposed NeuRoSFM for pedestrian trajectory prediction. The input to NeuRoSFM include pedestrian velocity, goal direction, distance and direction to other pedestrians, distance and direction to the robot, and direction to the group centroid. These inputs are provided to five different networks, each of which outputs the force experienced due to the individual component. The final outputs are combined to get the resulting social force acting on the pedestrian.