Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments

Hyunki Seong, David Hyunchul Shim

TL;DR

This work tackles mapless navigation in unknown environments by proposing Skill Q-Network (SQN), a reinforcement learning framework that simultaneously learns a high-level skill decision and multiple low-level navigation skills without requiring predefined priors. SQN uses a perception-planning architecture with latent skill policies and a softmax-based skill decision to ensemble Q-values, trained under a burn-in R2D2 framework to handle sequential observations. A tailored reward combines exploration and reachability signals, enabling effective navigation and avoiding local minima, with results showing up to about 40% performance gains over baselines and robust zero-shot transfer to unseen, out-of-distribution environments such as non-convex obstacles and cave-like terrains. The approach demonstrates strong adaptability and robustness, suggesting practical potential for real-world deployment and pointing to future work on dynamics-level versatility and sim-to-real transfer via domain randomization.

Abstract

This paper focuses on the acquisition of mapless navigation skills within unknown environments. We introduce the Skill Q-Network (SQN), a novel reinforcement learning method featuring an adaptive skill ensemble mechanism. Unlike existing methods, our model concurrently learns a high-level skill decision process alongside multiple low-level navigation skills, all without the need for prior knowledge. Leveraging a tailored reward function for mapless navigation, the SQN is capable of learning adaptive maneuvers that incorporate both exploration and goal-directed skills, enabling effective navigation in new environments. Our experiments demonstrate that our SQN can effectively navigate complex environments, exhibiting a 40\% higher performance compared to baseline models. Without explicit guidance, SQN discovers how to combine low-level skill policies, showcasing both goal-directed navigations to reach destinations and exploration maneuvers to escape from local minimum regions in challenging scenarios. Remarkably, our adaptive skill ensemble method enables zero-shot transfer to out-of-distribution domains, characterized by unseen observations from non-convex obstacles or uneven, subterranean-like environments. The project page is available at https://sites.google.com/view/skill-q-net.

Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments

TL;DR

This work tackles mapless navigation in unknown environments by proposing Skill Q-Network (SQN), a reinforcement learning framework that simultaneously learns a high-level skill decision and multiple low-level navigation skills without requiring predefined priors. SQN uses a perception-planning architecture with latent skill policies and a softmax-based skill decision to ensemble Q-values, trained under a burn-in R2D2 framework to handle sequential observations. A tailored reward combines exploration and reachability signals, enabling effective navigation and avoiding local minima, with results showing up to about 40% performance gains over baselines and robust zero-shot transfer to unseen, out-of-distribution environments such as non-convex obstacles and cave-like terrains. The approach demonstrates strong adaptability and robustness, suggesting practical potential for real-world deployment and pointing to future work on dynamics-level versatility and sim-to-real transfer via domain randomization.

Abstract

This paper focuses on the acquisition of mapless navigation skills within unknown environments. We introduce the Skill Q-Network (SQN), a novel reinforcement learning method featuring an adaptive skill ensemble mechanism. Unlike existing methods, our model concurrently learns a high-level skill decision process alongside multiple low-level navigation skills, all without the need for prior knowledge. Leveraging a tailored reward function for mapless navigation, the SQN is capable of learning adaptive maneuvers that incorporate both exploration and goal-directed skills, enabling effective navigation in new environments. Our experiments demonstrate that our SQN can effectively navigate complex environments, exhibiting a 40\% higher performance compared to baseline models. Without explicit guidance, SQN discovers how to combine low-level skill policies, showcasing both goal-directed navigations to reach destinations and exploration maneuvers to escape from local minimum regions in challenging scenarios. Remarkably, our adaptive skill ensemble method enables zero-shot transfer to out-of-distribution domains, characterized by unseen observations from non-convex obstacles or uneven, subterranean-like environments. The project page is available at https://sites.google.com/view/skill-q-net.
Paper Structure (23 sections, 6 equations, 8 figures, 1 table)

This paper contains 23 sections, 6 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Conventional Mapless Navigation
  4. Learning-based Mapless Navigation
  5. Methodologies
  6. State and Action Representation
  7. Skill Q-Network
  8. Reward Function
  9. Experiments
  10. Environments Setup
  11. Robustness Settings
  12. Hyperparameter Configuration
  13. Baseline Models
  14. Quantitative Evaluation
  15. Learning Curves
  16. ...and 8 more sections

Figures (8)

  • Figure 1: Overview of the Skill Q-Network.
  • Figure 2: Detailed network architecture of the Skill Q-Network.
  • Figure 3: An example illustrating the map coverage, along with the corresponding status and observation of the ego agent.
  • Figure 4: Examples of goal (non-)reachability. In mapless environments, situations frequently arise where the goal point is close in terms of Euclidean distance yet remains unreachable.
  • Figure 5: Various environments and observation patterns for experiments. Observations are visualized with the ego agent's center point as the origin, where the local x-axis points east.
  • ...and 3 more figures