Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Path-Planning for Autonomous Robots: A UCH-Enhanced Q-Learning Approach

Wei Liu, Ruiyang Wang, Haonan Wang, Guangwei Liu

TL;DR

This work addresses slow convergence and local optima in Q-learning for robot path planning by introducing the Improved Q-learning (IQL) framework. It fuses a Path Adaptive Collaborative Optimization (PACO) for smarter Q-table initialization with a Utility-Controlled Heuristic (UCH) for dynamic reward shaping, including distance-based metric considerations. Empirical results on raster maps of varying sizes show that IQL achieves faster convergence, higher stability, and improved path quality compared to several Q-learning variants, with notable gains using Chebyshev distance. The approach promises more efficient and reliable autonomous navigation in complex environments, offering practical impact for real-time robotic path planning.

Abstract

Q-learning methods are widely used in robot path planning but often face challenges of inefficient search and slow convergence. We propose an Improved Q-learning (IQL) framework that enhances standard Q-learning in two significant ways. First, we introduce the Path Adaptive Collaborative Optimization (PACO) algorithm to optimize Q-table initialization, providing better initial estimates and accelerating learning. Second, we incorporate a Utility-Controlled Heuristic (UCH) mechanism with dynamically tuned parameters to optimize the reward function, enhancing the algorithm's accuracy and effectiveness in path-planning tasks. Extensive experiments in three different raster grid environments validate the superior performance of our IQL framework. The results demonstrate that our IQL algorithm outperforms existing methods, including FIQL, PP-QL-based CPP, DFQL, and QMABC algorithms, in terms of path-planning capabilities.

Adaptive Path-Planning for Autonomous Robots: A UCH-Enhanced Q-Learning Approach

TL;DR

This work addresses slow convergence and local optima in Q-learning for robot path planning by introducing the Improved Q-learning (IQL) framework. It fuses a Path Adaptive Collaborative Optimization (PACO) for smarter Q-table initialization with a Utility-Controlled Heuristic (UCH) for dynamic reward shaping, including distance-based metric considerations. Empirical results on raster maps of varying sizes show that IQL achieves faster convergence, higher stability, and improved path quality compared to several Q-learning variants, with notable gains using Chebyshev distance. The approach promises more efficient and reliable autonomous navigation in complex environments, offering practical impact for real-time robotic path planning.

Abstract

Q-learning methods are widely used in robot path planning but often face challenges of inefficient search and slow convergence. We propose an Improved Q-learning (IQL) framework that enhances standard Q-learning in two significant ways. First, we introduce the Path Adaptive Collaborative Optimization (PACO) algorithm to optimize Q-table initialization, providing better initial estimates and accelerating learning. Second, we incorporate a Utility-Controlled Heuristic (UCH) mechanism with dynamically tuned parameters to optimize the reward function, enhancing the algorithm's accuracy and effectiveness in path-planning tasks. Extensive experiments in three different raster grid environments validate the superior performance of our IQL framework. The results demonstrate that our IQL algorithm outperforms existing methods, including FIQL, PP-QL-based CPP, DFQL, and QMABC algorithms, in terms of path-planning capabilities.
Paper Structure (17 sections, 9 equations, 13 figures, 14 tables)

This paper contains 17 sections, 9 equations, 13 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning
  4. Basic Q-learning algorithms
  5. Environmental modelling for path planning
  6. Methods
  7. Q-table initialization optimization strategy
  8. PACO algorithm
  9. Q-table initialization operation
  10. Reward function optimization
  11. Evaluation indicators
  12. Experiments
  13. Simulation environment
  14. Algorithm validation
  15. Algorithm comparison
  16. ...and 2 more sections

Figures (13)

  • Figure 1: Framework diagram of the improved Q-learning (IQL) algorithm
  • Figure 2: Interplay process of the Q-learning algorithm and the surrounding environment
  • Figure 3: Raster map environment and serial number encoding
  • Figure 4: The selected path and the correct detour path
  • Figure 5: Two error paths
  • ...and 8 more figures