Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving robot navigation in crowded environments using intrinsic rewards

Diego Martinez-Baselga, Luis Riazuelo, Luis Montano

TL;DR

This paper tackles robot navigation in crowded environments by addressing exploration challenges in deep reinforcement learning. It introduces two intrinsic reward schemes, Intrinsic Curiosity Module (ICM) and Random Encoders for Efficient Exploration (RE3), to steer learning towards uncertain states and avoid local optima. Across CrowdNav-based experiments and multiple DRL architectures, intrinsic rewards consistently yield faster learning, higher rewards, and safer, shorter navigation times, with ICM often providing the strongest gains. The findings suggest intrinsic motivation as a practical tool to enhance real-time crowd navigation and potentially extend to other dynamic-avoidance tasks in smart-city contexts.

Abstract

Autonomous navigation in crowded environments is an open problem with many applications, essential for the coexistence of robots and humans in the smart cities of the future. In recent years, deep reinforcement learning approaches have proven to outperform model-based algorithms. Nevertheless, even though the results provided are promising, the works are not able to take advantage of the capabilities that their models offer. They usually get trapped in local optima in the training process, that prevent them from learning the optimal policy. They are not able to visit and interact with every possible state appropriately, such as with the states near the goal or near the dynamic obstacles. In this work, we propose using intrinsic rewards to balance between exploration and exploitation and explore depending on the uncertainty of the states instead of on the time the agent has been trained, encouraging the agent to get more curious about unknown states. We explain the benefits of the approach and compare it with other exploration algorithms that may be used for crowd navigation. Many simulation experiments are performed modifying several algorithms of the state-of-the-art, showing that the use of intrinsic rewards makes the robot learn faster and reach higher rewards and success rates (fewer collisions) in shorter navigation times, outperforming the state-of-the-art.

Improving robot navigation in crowded environments using intrinsic rewards

TL;DR

This paper tackles robot navigation in crowded environments by addressing exploration challenges in deep reinforcement learning. It introduces two intrinsic reward schemes, Intrinsic Curiosity Module (ICM) and Random Encoders for Efficient Exploration (RE3), to steer learning towards uncertain states and avoid local optima. Across CrowdNav-based experiments and multiple DRL architectures, intrinsic rewards consistently yield faster learning, higher rewards, and safer, shorter navigation times, with ICM often providing the strongest gains. The findings suggest intrinsic motivation as a practical tool to enhance real-time crowd navigation and potentially extend to other dynamic-avoidance tasks in smart-city contexts.

Abstract

Autonomous navigation in crowded environments is an open problem with many applications, essential for the coexistence of robots and humans in the smart cities of the future. In recent years, deep reinforcement learning approaches have proven to outperform model-based algorithms. Nevertheless, even though the results provided are promising, the works are not able to take advantage of the capabilities that their models offer. They usually get trapped in local optima in the training process, that prevent them from learning the optimal policy. They are not able to visit and interact with every possible state appropriately, such as with the states near the goal or near the dynamic obstacles. In this work, we propose using intrinsic rewards to balance between exploration and exploitation and explore depending on the uncertainty of the states instead of on the time the agent has been trained, encouraging the agent to get more curious about unknown states. We explain the benefits of the approach and compare it with other exploration algorithms that may be used for crowd navigation. Many simulation experiments are performed modifying several algorithms of the state-of-the-art, showing that the use of intrinsic rewards makes the robot learn faster and reach higher rewards and success rates (fewer collisions) in shorter navigation times, outperforming the state-of-the-art.
Paper Structure (14 sections, 7 equations, 4 figures, 1 table)

This paper contains 14 sections, 7 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Crowd navigation
  4. Exploration in Deep Reinforcement Learning
  5. Problem formulation
  6. Approach
  7. Exploration problem
  8. Intrinsic reward
  9. Experiments
  10. Environment settings
  11. SG-D3QN
  12. Other models
  13. Conclusion
  14. Introduction

Figures (4)

  • Figure 1: Comparison shows colliding (left) vs. safe (right) trajectory, obtained by same algorithm without/with intrinsic rewards.
  • Figure 2: A diagram of the Intrinsic Curiosity Module.
  • Figure 3: Comparison of robot trajectories using $\epsilon$-greedy and ICM exploration at different stages of training (500, 2500, 5000 episodes and final model).
  • Figure 4: Accumulated reward, success rate and navigation time metrics gathered during training, with a smoothing factor of 0.99.