Walk the PLANC: Physics-Guided RL for Agile Humanoid Locomotion on Constrained Footholds

TL;DR

This work tackles the challenge of agile humanoid locomotion on constrained footholds by marrying a reduced-order stepping planner with reinforcement learning through CLF-based rewards. The approach generates dynamically consistent references (foot placement, CoM, timing) from a LIP-based model and uses a teacher-student distillation paradigm to bootstrap a deployable policy that tracks these references under partial observability. The method achieves superior reliability and precision on stepping-stone terrains, demonstrated in simulation and transferred to real hardware (Unitree G1) with robust sim-to-real performance and zero-shot generalization to novel terrain. By combining model-based structure with data-driven adaptation, the framework reduces sample complexity and improves transferability while maintaining agile, accurate locomotion on discrete footholds.

Abstract

Bipedal humanoid robots must precisely coordinate balance, timing, and contact decisions when locomoting on constrained footholds such as stepping stones, beams, and planks -- even minor errors can lead to catastrophic failure. Classical optimization and control pipelines handle these constraints well but depend on highly accurate mathematical representations of terrain geometry, making them prone to error when perception is noisy or incomplete. Meanwhile, reinforcement learning has shown strong resilience to disturbances and modeling errors, yet end-to-end policies rarely discover the precise foothold placement and step sequencing required for discontinuous terrain. These contrasting limitations motivate approaches that guide learning with physics-based structure rather than relying purely on reward shaping. In this work, we introduce a locomotion framework in which a reduced-order stepping planner supplies dynamically consistent motion targets that steer the RL training process via Control Lyapunov Function (CLF) rewards. This combination of structured footstep planning and data-driven adaptation produces accurate, agile, and hardware-validated stepping-stone locomotion on a humanoid robot, substantially improving reliability compared to conventional model-free reinforcement-learning baselines.

Figures (8)

• Figure 1: Model-guided RL traversing constrained footholds on the Unitree G1 humanoid robot. The policy is trained with environmentally consistent references generated dynamically during training by a momentum-regulated linear inverted pendulum for precise footstepping and center of mass regulation, and is deployed successfully on hardware.
• Figure 2: A visual depiction of the model-guided RL architecture used to achieve stepping stones. The left column shows the four terrains trained in simulation, (top to bottom) stairs up, stairs down, flat stones, and height varying stones. Training is carried out in a teach-student fashion, with CLF rewards informed by the RoM. The student policy is successfully deploy in sim-to-sim transfer and on hardware.
• Figure 3: A visual depiction of the LIP model used for gait synthesis, including the desired trajectories for the swing foot $\boldsymbol{p}_{\text{sw}}$ and CoM $\boldsymbol{p}_{\text{com}}$.
• Figure 4: (Left) The CLF reward curves during student policy training for the model-guided policies. The proposed method obtains better tracking than both the Fixed $\dot{z}_{\text{com}}$ and Fixed $T$ methods. (Right) A comparison between the model-guided policies and a naive policy; the naive policy is not able to handle the more difficult terrains.
• Figure 5: The swing foot (left) and CoM (right) tracking in the $x$ direction. Precise foot tracking is critical to hit constrained footholds, and CoM tracking is particularly good around contact, improving performance on future stones.