Reinforcement Learning for Adaptive Planner Parameter Tuning: A Perspective on Hierarchical Architecture

TL;DR

The paper tackles adaptive planner parameter tuning for robotic navigation in constrained environments by introducing a three-layer hierarchical RL framework with low-frequency tuning ($1\ \mathrm{Hz}$), mid-frequency planning ($10\ \mathrm{Hz}$), and high-frequency control ($50\ \mathrm{Hz}$), augmented by an RL-based error compensator and LiDAR-based obstacle avoidance. It formulates an alternating training scheme using TD3 to jointly improve tuning and control, and demonstrates superior performance in the BARN benchmark with strong sim-to-real transfer. Key findings show that carefully separating frequencies and iteratively refining tuning and control yield high success rates (up to $98\%$) and reduced completion times, while ablations highlight the benefits of a feedback-only controller and feature-based state representations. The results suggest a practical pathway for integrating traditional planning guarantees with learning-based adaptation to achieve robust navigation in real-world settings.

Abstract

Automatic parameter tuning methods for planning algorithms, which integrate pipeline approaches with learning-based techniques, are regarded as promising due to their stability and capability to handle highly constrained environments. While existing parameter tuning methods have demonstrated considerable success, further performance improvements require a more structured approach. In this paper, we propose a hierarchical architecture for reinforcement learning-based parameter tuning. The architecture introduces a hierarchical structure with low-frequency parameter tuning, mid-frequency planning, and high-frequency control, enabling concurrent enhancement of both upper-layer parameter tuning and lower-layer control through iterative training. Experimental evaluations in both simulated and real-world environments show that our method surpasses existing parameter tuning approaches. Furthermore, our approach achieves first place in the Benchmark for Autonomous Robot Navigation (BARN) Challenge.

Figures (9)

• Figure 1: Different frameworks for parameter tuning. (a): Existing framework. (b): Asynchronous hierarchical architecture.
• Figure 2: Illustration of the hierarchical architecture. The parameter tuning network selects appropriate parameters for the local planner, which adjusts these parameters to generate both the trajectory and feedforward velocity. The controller then calculates the feedback velocity using LiDAR data, robot state, the planned trajectory and time step. The final speed control command is produced by combining the feedback and feedforward velocities.
• Figure 3: Illustration of alternating training of the hierarchical architecture. FC Controller represents feedforward controller. RL Controller represents the proposed controller.
• Figure 4: The detailed architecture of the Actor-Critic network. The input laser scan data is encoded into a latent space using a VAE. The rest of the network consists of Multi-Layer Perceptron (MLP).
• Figure 5: Comparison of training loss curves. The local planner operates at 10 Hz, and we compare the training loss between 1 Hz and 10 Hz parameter tuning.