Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning

TL;DR

The paper tackles robust quadruped navigation in environments with dynamic obstacles by proposing Hi-QARL, a hierarchical controller trained against adversarial obstacles whose rationality is bounded via quantal response equilibria and gradually increased through a self-paced curriculum. The approach combines a two-level policy (low-level locomotion via PPO and high-level navigation via SAC) with an adversarial obstacle modeled as a moving agent, whose temperature parameter is annealed to produce a curriculum of increasing difficulty. Empirical results in simulated unitree GO1 experiments show that Hi-QARL improves robustness to unseen obstacles compared to baselines like RARL, with observable curriculum effects in adversary entropy and temperature trajectories. The work demonstrates practical gains in dynamic obstacle avoidance and offers a pathway toward more reliable sim-to-real deployment, while acknowledging current limitations to single obstacle scenarios and suggesting future generalizations to multiple moving obstacles.

Abstract

Reinforcement Learning (RL) has proven largely effective in obtaining stable locomotion gaits for legged robots. However, designing control algorithms which can robustly navigate unseen environments with obstacles remains an ongoing problem within quadruped locomotion. To tackle this, it is convenient to solve navigation tasks by means of a hierarchical approach with a low-level locomotion policy and a high-level navigation policy. Crucially, the high-level policy needs to be robust to dynamic obstacles along the path of the agent. In this work, we propose a novel way to endow navigation policies with robustness by a training process that models obstacles as adversarial agents, following the adversarial RL paradigm. Importantly, to improve the reliability of the training process, we bound the rationality of the adversarial agent resorting to quantal response equilibria, and place a curriculum over its rationality. We called this method Hierarchical policies via Quantal response Adversarial Reinforcement Learning (Hi-QARL). We demonstrate the robustness of our method by benchmarking it in unseen randomized mazes with multiple obstacles. To prove its applicability in real scenarios, our method is applied on a Unitree GO1 robot in simulation.

Figures (6)

• Figure 1: Sequence of a robot Unitree GO1 using Hi-QARL to navigate a room with a moving obstacle (yellow box) towards a target position (red dot).
• Figure 2: Online success rate of RARL vs Hi-QARL.
• Figure 3: Adversarial policy entropy of RARL and Hi-QARL during training.
• Figure 4: Mean temperature of Hi-QARL adversary during training.
• Figure 5: GO1 robot successfully navigates a new maze with two new unseen static obstacles in addition to a moving obstacle.