Robust Humanoid Walking on Compliant and Uneven Terrain with Deep Reinforcement Learning

TL;DR

This paper tackles robust humanoid locomotion on compliant and uneven terrains using sim-to-real deep reinforcement learning. It introduces a two-stage curriculum that randomizes terrain properties and robot dynamics in simulation, while relying on proprioceptive feedback and a clock-based input to train a unified bipedal policy. A key contribution is the clock-modulation (aperiodic) gait policy, which shows improved robustness in simulation and enables longer, safer walking on challenging surfaces, with real-world HRP-5P demonstrations achieving meaningful mobility indoors and outdoors. The work advances practical, transferable locomotion for life-sized humanoids and provides open-source code to facilitate reproducibility and further research.

Abstract

For the deployment of legged robots in real-world environments, it is essential to develop robust locomotion control methods for challenging terrains that may exhibit unexpected deformability and irregularity. In this paper, we explore the application of sim-to-real deep reinforcement learning (RL) for the design of bipedal locomotion controllers for humanoid robots on compliant and uneven terrains. Our key contribution is to show that a simple training curriculum for exposing the RL agent to randomized terrains in simulation can achieve robust walking on a real humanoid robot using only proprioceptive feedback. We train an end-to-end bipedal locomotion policy using the proposed approach, and show extensive real-robot demonstration on the HRP-5P humanoid over several difficult terrains inside and outside the lab environment. Further, we argue that the robustness of a bipedal walking policy can be improved if the robot is allowed to exhibit aperiodic motion with variable stepping frequency. We propose a new control policy to enable modification of the observed clock signal, leading to adaptive gait frequencies depending on the terrain and command velocity. Through simulation experiments, we show the effectiveness of this policy specifically for walking over challenging terrains by controlling swing and stance durations. The code for training and evaluation is available online at https://github.com/rohanpsingh/LearningHumanoidWalking. Demo video is available at https://www.youtube.com/watch?v=ZgfNzGAkk2Q.

Figures (4)

• Figure 1: HRP-5P humanoid bipedal locomotion (clockwise) on flat rigid floor, soft cushion, uneven inclined blocks, paved street, and grass using learned policy. The same RL policy was used for all terrains without any parameter tuning in between experiments. Lifter and overhead crane serve as a failsafe, the ropes are slack and robot is not provided external support.
• Figure 2: Overview of our training framework. (L) We propose to train a feedforward RL agent while exposing it to randomized dynamics parameters in the first phase and then, additionally, randomized uneven and compliant terrains in the second phase. The policy achieves zero-shot sim-to-real transfer on the real HRP-5P. (R) We also propose an augmented policy that can make clock signal modifications for regulating the stepping frequency to achieve improved robustness on challenging terrains.
• Figure 3: Behavior analysis of Clock Control (in simulation). (a) Training reward curves for the clock-control policy and the default policy averaged over 3 training sessions with separate random seeds. Both policies are trained on randomized terrains starting from a regular terrain pre-trained policy. The clock-control policy converges to a higher reward. (b) 1-minute long episodes on the same uneven and soft terrain. The clock-control policy modifies the swing and stance duration in response to large disturbances while the default policy maintains a fixed gait pattern. Note. The sharp peak force for clock-control is due to inaccuracy in simulation (contact penetration).
• Figure 4: Evolution of the phase variable for a 20-second episode rollout on flat, rigid terrain for walking forward and stepping in-place with a clock-control policy. Policy phase offset predictions (top) and the actual scalar phase variable (bottom) are both normalized by a fixed period $L = 80$ for better visualization.