Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficiently Learning Robust Torque-based Locomotion Through Reinforcement with Model-Based Supervision

Yashuai Yan, Tobias Egle, Christian Ott, Dongheui Lee

TL;DR

This work tackles the sim-to-real gap and data inefficiency in torque-based bipedal locomotion by fusing a physics-based base controller with a residual reinforcement learning policy trained under domain randomization. A privileged Oracle policy supervises the residual learner through a dedicated loss, accelerating learning and improving robustness without heavy reward shaping. The approach leverages a DCM-based trajectory generator and an inverse dynamics WBC, with a PPO-based residual policy guided by both RL signals and Oracle corrections. Evaluations across Kangaroo, Unitree H1-2, and Bruce demonstrate near-Oracle performance, strong terrain robustness, and transferability without robot-specific tuning, highlighting the method’s potential for scalable sim-to-real transfer in legged robots.

Abstract

We propose a control framework that integrates model-based bipedal locomotion with residual reinforcement learning (RL) to achieve robust and adaptive walking in the presence of real-world uncertainties. Our approach leverages a model-based controller, comprising a Divergent Component of Motion (DCM) trajectory planner and a whole-body controller, as a reliable base policy. To address the uncertainties of inaccurate dynamics modeling and sensor noise, we introduce a residual policy trained through RL with domain randomization. Crucially, we employ a model-based oracle policy, which has privileged access to ground-truth dynamics during training, to supervise the residual policy via a novel supervised loss. This supervision enables the policy to efficiently learn corrective behaviors that compensate for unmodeled effects without extensive reward shaping. Our method demonstrates improved robustness and generalization across a range of randomized conditions, offering a scalable solution for sim-to-real transfer in bipedal locomotion.

Efficiently Learning Robust Torque-based Locomotion Through Reinforcement with Model-Based Supervision

TL;DR

This work tackles the sim-to-real gap and data inefficiency in torque-based bipedal locomotion by fusing a physics-based base controller with a residual reinforcement learning policy trained under domain randomization. A privileged Oracle policy supervises the residual learner through a dedicated loss, accelerating learning and improving robustness without heavy reward shaping. The approach leverages a DCM-based trajectory generator and an inverse dynamics WBC, with a PPO-based residual policy guided by both RL signals and Oracle corrections. Evaluations across Kangaroo, Unitree H1-2, and Bruce demonstrate near-Oracle performance, strong terrain robustness, and transferability without robot-specific tuning, highlighting the method’s potential for scalable sim-to-real transfer in legged robots.

Abstract

We propose a control framework that integrates model-based bipedal locomotion with residual reinforcement learning (RL) to achieve robust and adaptive walking in the presence of real-world uncertainties. Our approach leverages a model-based controller, comprising a Divergent Component of Motion (DCM) trajectory planner and a whole-body controller, as a reliable base policy. To address the uncertainties of inaccurate dynamics modeling and sensor noise, we introduce a residual policy trained through RL with domain randomization. Crucially, we employ a model-based oracle policy, which has privileged access to ground-truth dynamics during training, to supervise the residual policy via a novel supervised loss. This supervision enables the policy to efficiently learn corrective behaviors that compensate for unmodeled effects without extensive reward shaping. Our method demonstrates improved robustness and generalization across a range of randomized conditions, offering a scalable solution for sim-to-real transfer in bipedal locomotion.
Paper Structure (25 sections, 9 equations, 5 figures, 5 tables, 1 algorithm)

This paper contains 25 sections, 9 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model-based Robot Control
  4. DCM Trajectory Generation
  5. Whole-body Robot Control
  6. Enhance Robustness through Randomization
  7. Model-based Supervision
  8. Supervised Reinforcement Learning
  9. Observation Space
  10. Reward Functions
  11. Supervised Loss
  12. Evaluation
  13. Technical Implementation
  14. Baselines
  15. Model-based Controller (MBC)
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Residual Reinforcement Learning with Model-Based Supervision. Our framework integrates three key components: the Base Policy, the Oracle Policy, and the Residual Policy. The Base and Oracle policies are model-based controllers; however, the Oracle, only used in training, has privileged access to true system information, including the robot model, state, and motor parameters, while the Base Policy operates under an inaccurate dynamics model with realistic assumptions without access to the true system information (LP: low-pass filter). The learnable Residual Policy is trained to compensate for the model inaccuracies of the Base Policy. Crucially, the Residual Policy is guided by both the RL objective $\mathcal{L}_{\text{rl}}$ and direct supervision from the Oracle Policy $\mathcal{L}_{\text{sup}}$, enabling efficient learning and improved robustness under real-world uncertainties.
  • Figure 2: Quantitative evaluation on the Kangaroo robot.We evaluate various methods using predefined metrics throughout training. Our method (BOR) shows that residual learning converges substantially faster than directly learning torque commands, even under identical Oracle supervision. Moreover, comparing BOR/OR with other baselines demonstrates that incorporating the supervision term into the optimization objective significantly improves training efficiency, bringing performance much closer to that of the Oracle policies.
  • Figure 3: Torque tracking on the right hip pitch joint. The Base policy produces noisy torques due to the domain randomization. Our learned residual policy succeeds in compensating the joint torques, closely tracking the Oracle policy.
  • Figure 4: Quantitative evaluation on DCM and foot tracking. The H1-2 robot walks straight forward at a speed of $0.2 m/s$, while the Kangaroo is commanded with a linear velocity of $0.2 m/s$ and an angular velocity of $0.2 rad/s$. $\xi_{des}$ shows the planned DCM trajectory and the footprints are visualized as polygons in green and orange.
  • Figure 5: Evaluation on uneven terrain. Uneven terrain introduces additional uncertainties in real-world environments, yet even the Oracle policy struggles to explicitly encode ground irregularities. Thanks to the learning paradigms in our framework, our method learns adaptive behaviors that go beyond simply mimicking the Oracle, unlike the baseline IL approach.