Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Generalization in Reinforcement Learning Training Regimes for Social Robot Navigation

Adam Sigal, Hsiu-Chin Lin, AJung Moon

TL;DR

The paper tackles the generalization gap in RL-based social navigation by showing that training in overly simple environments with homogeneous pedestrian behavior fails to transfer to realistic, crowded spaces. It introduces curriculum learning and diversified pedestrian dynamics (ORCA and Social Force) to train three RL models (CADRL, LSTM-RL, SARL) under four regimes, then evaluates them in larger, unseen Diverse-4 environments that simulate complex crowds. The study finds that the curriculum plus diversity approach, particularly when combined with SARL (CD-SARL), yields the best generalization, reducing pedestrian discomfort while maintaining reasonable efficiency and safety. These findings challenge prior work that reports strong results only in simplistic settings and underscore the need for richer training curricula and out-of-distribution evaluation for social robot navigation.

Abstract

In order for autonomous mobile robots to navigate in human spaces, they must abide by our social norms. Reinforcement learning (RL) has emerged as an effective method to train sequential decision-making policies that are able to respect these norms. However, a large portion of existing work in the field conducts both RL training and testing in simplistic environments. This limits the generalization potential of these models to unseen environments, and the meaningfulness of their reported results. We propose a method to improve the generalization performance of RL social navigation methods using curriculum learning. By employing multiple environment types and by modeling pedestrians using multiple dynamics models, we are able to progressively diversify and escalate difficulty in training. Our results show that the use of curriculum learning in training can be used to achieve better generalization performance than previous training methods. We also show that results presented in many existing state-of-the-art RL social navigation works do not evaluate their methods outside of their training environments, and thus do not reflect their policies' failure to adequately generalize to out-of-distribution scenarios. In response, we validate our training approach on larger and more crowded testing environments than those used in training, allowing for more meaningful measurements of model performance.

Improving Generalization in Reinforcement Learning Training Regimes for Social Robot Navigation

TL;DR

The paper tackles the generalization gap in RL-based social navigation by showing that training in overly simple environments with homogeneous pedestrian behavior fails to transfer to realistic, crowded spaces. It introduces curriculum learning and diversified pedestrian dynamics (ORCA and Social Force) to train three RL models (CADRL, LSTM-RL, SARL) under four regimes, then evaluates them in larger, unseen Diverse-4 environments that simulate complex crowds. The study finds that the curriculum plus diversity approach, particularly when combined with SARL (CD-SARL), yields the best generalization, reducing pedestrian discomfort while maintaining reasonable efficiency and safety. These findings challenge prior work that reports strong results only in simplistic settings and underscore the need for richer training curricula and out-of-distribution evaluation for social robot navigation.

Abstract

In order for autonomous mobile robots to navigate in human spaces, they must abide by our social norms. Reinforcement learning (RL) has emerged as an effective method to train sequential decision-making policies that are able to respect these norms. However, a large portion of existing work in the field conducts both RL training and testing in simplistic environments. This limits the generalization potential of these models to unseen environments, and the meaningfulness of their reported results. We propose a method to improve the generalization performance of RL social navigation methods using curriculum learning. By employing multiple environment types and by modeling pedestrians using multiple dynamics models, we are able to progressively diversify and escalate difficulty in training. Our results show that the use of curriculum learning in training can be used to achieve better generalization performance than previous training methods. We also show that results presented in many existing state-of-the-art RL social navigation works do not evaluate their methods outside of their training environments, and thus do not reflect their policies' failure to adequately generalize to out-of-distribution scenarios. In response, we validate our training approach on larger and more crowded testing environments than those used in training, allowing for more meaningful measurements of model performance.
Paper Structure (14 sections, 3 equations, 3 figures, 2 tables)

This paper contains 14 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. Previous Work
  4. Social Force and ORCA
  5. Reinforcement Learning for Social Navigation
  6. METHODOLOGY
  7. EXPERIMENTS
  8. Implementation Details
  9. Evaluation
  10. Simulator
  11. Comparison of Evaluation Environments
  12. Quantitative Ablation Study of Training Techniques
  13. Qualitative Experiments
  14. CONCLUSION

Figures (3)

  • Figure 1: Examples of social navigation scenarios in the simple setting. White, numbered circles represent pedestrians, while the yellow circle represents the robot. The circular and squared outlines in the figures are not environment borders; they simply serve to illustrate the experimental environment type. All 6 environment types are described in Table \ref{['tab:env-types']}.
  • Figure 2: Evaluation environment comparison. Average success rate of baseline models (baseline training) on original evaluation environment (ORCA-only small circle crossing) vs. on the more challenging Diverse-4 evaluation environments. Our best method, CD-SARL, generalizes better to the Diverse-4 environments, with a 0.95 success rate (see Table \ref{['tab:ablat']}).
  • Figure 3: (a) Scene in a diverse, dense square crossing after 2.25 seconds of navigation. Value estimations of each robot agent's action space in this scene (direction, and 5 increments of speed between 0 and $v_{pref}$) by BL-SARL (b) and CD-SARL (c). Values are shown on a gradient, where the highest estimated values are yellow, and the lowest are blue.