Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

End-to-End Reinforcement Learning for Torque Based Variable Height Hopping

Raghav Soni, Daniel Harnack, Hannah Isermann, Sotaro Fushimi, Shivesh Kumar, Frank Kirchner

TL;DR

This work tackles dynamic, height-adjustable hopping with a monoped by training an end-to-end torque controller that maps proprioceptive feedback directly to joint torques, thereby avoiding explicit jump-phase detection and PD-based control strategies. It combines an energy-shaping baseline as a reference with a Soft Actor-Critic (SAC) policy trained in MuJoCo simulation, using a carefully designed reward that includes an energy term and penalties for height excess, jerky actions, and joint constraints. A two-stage CMA-ES based sim-to-real parameter optimization is employed to bridge the gap between simulation and the real robot, supplemented by robustness strategies for delays and sensor/actuator noise. The resulting controller demonstrates interpolation across jump heights and successful transfer to hardware without parameter tuning, marking a first end-to-end monoped hopping controller, with practical implications for adaptable, torque-based locomotion on unstructured terrains.

Abstract

Legged locomotion is arguably the most suited and versatile mode to deal with natural or unstructured terrains. Intensive research into dynamic walking and running controllers has recently yielded great advances, both in the optimal control and reinforcement learning (RL) literature. Hopping is a challenging dynamic task involving a flight phase and has the potential to increase the traversability of legged robots. Model based control for hopping typically relies on accurate detection of different jump phases, such as lift-off or touch down, and using different controllers for each phase. In this paper, we present a end-to-end RL based torque controller that learns to implicitly detect the relevant jump phases, removing the need to provide manual heuristics for state detection. We also extend a method for simulation to reality transfer of the learned controller to contact rich dynamic tasks, resulting in successful deployment on the robot after training without parameter tuning.

End-to-End Reinforcement Learning for Torque Based Variable Height Hopping

TL;DR

This work tackles dynamic, height-adjustable hopping with a monoped by training an end-to-end torque controller that maps proprioceptive feedback directly to joint torques, thereby avoiding explicit jump-phase detection and PD-based control strategies. It combines an energy-shaping baseline as a reference with a Soft Actor-Critic (SAC) policy trained in MuJoCo simulation, using a carefully designed reward that includes an energy term and penalties for height excess, jerky actions, and joint constraints. A two-stage CMA-ES based sim-to-real parameter optimization is employed to bridge the gap between simulation and the real robot, supplemented by robustness strategies for delays and sensor/actuator noise. The resulting controller demonstrates interpolation across jump heights and successful transfer to hardware without parameter tuning, marking a first end-to-end monoped hopping controller, with practical implications for adaptable, torque-based locomotion on unstructured terrains.

Abstract

Legged locomotion is arguably the most suited and versatile mode to deal with natural or unstructured terrains. Intensive research into dynamic walking and running controllers has recently yielded great advances, both in the optimal control and reinforcement learning (RL) literature. Hopping is a challenging dynamic task involving a flight phase and has the potential to increase the traversability of legged robots. Model based control for hopping typically relies on accurate detection of different jump phases, such as lift-off or touch down, and using different controllers for each phase. In this paper, we present a end-to-end RL based torque controller that learns to implicitly detect the relevant jump phases, removing the need to provide manual heuristics for state detection. We also extend a method for simulation to reality transfer of the learned controller to contact rich dynamic tasks, resulting in successful deployment on the robot after training without parameter tuning.
Paper Structure (26 sections, 18 equations, 9 figures, 1 table)

This paper contains 26 sections, 18 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Robotic System
  4. Energy Shaping
  5. Controller design
  6. Jumping height estimation
  7. Simulation and real system parameters
  8. Reinforcement Learning
  9. Problem Formulation
  10. Network architecture
  11. Observation and action space
  12. Reward
  13. Energy Gain ($G_e$)
  14. Height barrier penalty ($P_{h}$)
  15. Jerky Action Penalty ($P_j$)
  16. ...and 11 more sections

Figures (9)

  • Figure 1: RL based torque controlled jumping snapshots in simulation and on the real robot.
  • Figure 2: Comparison of control concepts.
  • Figure 3: Hopping leg used in the experiments.
  • Figure 4: State machine used for the energy shaping controller.
  • Figure 5: Parameter optimization pipeline.
  • ...and 4 more figures