Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HeR-DRL:Heterogeneous Relational Deep Reinforcement Learning for Decentralized Multi-Robot Crowd Navigation

Xinyu Zhou, Songhao Piao, Wenzheng Chi, Liguo Chen, Wei Li

TL;DR

HeR-DRL tackles decentralized multi-robot crowd navigation by explicitly modeling interaction heterogeneity with a robot-crowd heterogeneous graph and a two-layer heterogeneous GNN, producing state embeddings fed into a DRL policy with a discrete 80-action space. The method defines node types $\{v_{cr}, v_h, v_{or}\}$ and edge types $\{\mathcal{E}_{HORI}, \mathcal{E}_{HCRI}, \mathcal{E}_{CRORI}, \mathcal{E}_{HHI}, \mathcal{E}_{ORORI}\}$, and uses a shaped reward to optimize $Q^*(s_t,a_t)$ via a Deep Q-Network. Experiments show that HeR-DRL achieves superior safety and comfort across both single-robot and multi-robot circle-crossing tasks, with larger gains in multi-robot scenarios due to richer heterogeneous interactions, while maintaining competitive efficiency. The work highlights interaction heterogeneity as a key factor for robust crowd navigation and provides a public implementation to foster further development in decentralized, heterogeneous-agent navigation systems.

Abstract

Crowd navigation has received significant research attention in recent years, especially DRL-based methods. While single-robot crowd scenarios have dominated research, they offer limited applicability to real-world complexities. The heterogeneity of interaction among multiple agent categories, like in decentralized multi-robot pedestrian scenarios, are frequently disregarded. This "interaction blind spot" hinders generalizability and restricts progress towards robust navigation algorithms. In this paper, we propose a heterogeneous relational deep reinforcement learning(HeR-DRL), based on customised heterogeneous GNN, in order to improve navigation strategies in decentralized multi-robot crowd navigation. Firstly, we devised a method for constructing robot-crowd heterogenous relation graph that effectively simulates the heterogeneous pair-wise interaction relationships. We proposed a new heterogeneous graph neural network for transferring and aggregating the heterogeneous state information. Finally, we incorporate the encoded information into deep reinforcement learning to explore the optimal policy. HeR-DRL are rigorously evaluated through comparing it to state-of-the-art algorithms in both single-robot and multi-robot circle crowssing scenario. The experimental results demonstrate that HeR-DRL surpasses the state-of-the-art approaches in overall performance, particularly excelling in safety and comfort metrics. This underscores the significance of interaction heterogeneity for crowd navigation. The source code will be publicly released in https://github.com/Zhouxy-Debugging-Den/HeR-DRL.

HeR-DRL:Heterogeneous Relational Deep Reinforcement Learning for Decentralized Multi-Robot Crowd Navigation

TL;DR

HeR-DRL tackles decentralized multi-robot crowd navigation by explicitly modeling interaction heterogeneity with a robot-crowd heterogeneous graph and a two-layer heterogeneous GNN, producing state embeddings fed into a DRL policy with a discrete 80-action space. The method defines node types and edge types , and uses a shaped reward to optimize via a Deep Q-Network. Experiments show that HeR-DRL achieves superior safety and comfort across both single-robot and multi-robot circle-crossing tasks, with larger gains in multi-robot scenarios due to richer heterogeneous interactions, while maintaining competitive efficiency. The work highlights interaction heterogeneity as a key factor for robust crowd navigation and provides a public implementation to foster further development in decentralized, heterogeneous-agent navigation systems.

Abstract

Crowd navigation has received significant research attention in recent years, especially DRL-based methods. While single-robot crowd scenarios have dominated research, they offer limited applicability to real-world complexities. The heterogeneity of interaction among multiple agent categories, like in decentralized multi-robot pedestrian scenarios, are frequently disregarded. This "interaction blind spot" hinders generalizability and restricts progress towards robust navigation algorithms. In this paper, we propose a heterogeneous relational deep reinforcement learning(HeR-DRL), based on customised heterogeneous GNN, in order to improve navigation strategies in decentralized multi-robot crowd navigation. Firstly, we devised a method for constructing robot-crowd heterogenous relation graph that effectively simulates the heterogeneous pair-wise interaction relationships. We proposed a new heterogeneous graph neural network for transferring and aggregating the heterogeneous state information. Finally, we incorporate the encoded information into deep reinforcement learning to explore the optimal policy. HeR-DRL are rigorously evaluated through comparing it to state-of-the-art algorithms in both single-robot and multi-robot circle crowssing scenario. The experimental results demonstrate that HeR-DRL surpasses the state-of-the-art approaches in overall performance, particularly excelling in safety and comfort metrics. This underscores the significance of interaction heterogeneity for crowd navigation. The source code will be publicly released in https://github.com/Zhouxy-Debugging-Den/HeR-DRL.
Paper Structure (22 sections, 8 equations, 5 figures, 2 tables)

This paper contains 22 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. RELATED WORKS
  3. METHOD
  4. State Information Preprocessing
  5. State Parameterisation
  6. State Embedding
  7. Heterogeneous Interactive Relationship Representation
  8. Robot-Crowd Heterogeneous Graph Construction
  9. Heterogeneous Messaging and Aggregation
  10. Deep Reinforcement Learning for Crowd Navigation
  11. EXPERIMENTS
  12. Simulation Setup
  13. Simulation Environment
  14. Baselines and Ablation Models
  15. Evaluation Metrics
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Heterogeneous interaction relationships in decentralized multi-robot crowd navigation. The center robot is controlled by trained policy, while the other robots controlled by other unknown policies. Consequently, there are five heterogeneous pair-wise interactions: HHI refers to the interaction specifically occurring among human, HCRI refers to the interaction between human and the center robot, HORI refers to the interaction between human and other robots, CRORI refers to the interaction between the center robot and other robots, and ORORI refers to the interaction specifically occurring among other robot.
  • Figure 2: The overall framework of our HeR-DRL. Here is an illustration of a multi-robot crowd scenario with a 3H3O configuration.
  • Figure 3: Diagram of the nth-layer heterogeneous GNN with a 3H3O configuration.
  • Figure 4: Demonstration of our simulation environment. We mainly utilize two training and testing scenarios: (a) Single-robot circle crossing scenario; (b) Multi-robot circle crossing scenario.
  • Figure 5: The comparison of navigation trajectories in single-robot and multi-robot circle crossing.