Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control

Shuhao Liao, Peizhuo Li, Xinrong Yang, Linnan Chang, Zhaoxin Fan, Qing Wang, Lei Shi, Yuhong Cao, Wenjun Wu, Guillaume Sartoretti

TL;DR

Growing Policy Optimization (GPO) introduces a time-varying action transformation that gradually expands the executable action space during reinforcement learning for legged robots, addressing challenges in torque-based control. The framework preserves the PPO update structure while bounding gradient distortion, enabling stable early optimization and improved asymptotic performance. The authors provide theoretical bounds on gradient distortion, variance, and convergence, and demonstrate through simulation and hardware experiments that GPO yields faster learning and better final performance than fixed-action baselines on quadruped and hexapod platforms. This approach offers a general, environment-agnostic method to improve data efficiency and robustness in high-dimensional continuous control for legged locomotion.

Abstract

Training reinforcement learning (RL) policies for legged robots remains challenging due to high-dimensional continuous actions, hardware constraints, and limited exploration. Existing methods for locomotion and whole-body control work well for position-based control with environment-specific heuristics (e.g., reward shaping, curriculum design, and manual initialization), but are less effective for torque-based control, where sufficiently exploring the action space and obtaining informative gradient signals for training is significantly more difficult. We introduce Growing Policy Optimization (GPO), a training framework that applies a time-varying action transformation to restrict the effective action space in the early stage, thereby encouraging more effective data collection and policy learning, and then progressively expands it to enhance exploration and achieve higher expected return. We prove that this transformation preserves the PPO update rule and introduces only bounded, vanishing gradient distortion, thereby ensuring stable training. We evaluate GPO on both quadruped and hexapod robots, including zero-shot deployment of simulation-trained policies on hardware. Policies trained with GPO consistently achieve better performance. These results suggest that GPO provides a general, environment-agnostic optimization framework for learning legged locomotion.

GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control

TL;DR

Growing Policy Optimization (GPO) introduces a time-varying action transformation that gradually expands the executable action space during reinforcement learning for legged robots, addressing challenges in torque-based control. The framework preserves the PPO update structure while bounding gradient distortion, enabling stable early optimization and improved asymptotic performance. The authors provide theoretical bounds on gradient distortion, variance, and convergence, and demonstrate through simulation and hardware experiments that GPO yields faster learning and better final performance than fixed-action baselines on quadruped and hexapod platforms. This approach offers a general, environment-agnostic method to improve data efficiency and robustness in high-dimensional continuous control for legged locomotion.

Abstract

Training reinforcement learning (RL) policies for legged robots remains challenging due to high-dimensional continuous actions, hardware constraints, and limited exploration. Existing methods for locomotion and whole-body control work well for position-based control with environment-specific heuristics (e.g., reward shaping, curriculum design, and manual initialization), but are less effective for torque-based control, where sufficiently exploring the action space and obtaining informative gradient signals for training is significantly more difficult. We introduce Growing Policy Optimization (GPO), a training framework that applies a time-varying action transformation to restrict the effective action space in the early stage, thereby encouraging more effective data collection and policy learning, and then progressively expands it to enhance exploration and achieve higher expected return. We prove that this transformation preserves the PPO update rule and introduces only bounded, vanishing gradient distortion, thereby ensuring stable training. We evaluate GPO on both quadruped and hexapod robots, including zero-shot deployment of simulation-trained policies on hardware. Policies trained with GPO consistently achieve better performance. These results suggest that GPO provides a general, environment-agnostic optimization framework for learning legged locomotion.
Paper Structure (39 sections, 76 equations, 6 figures, 4 tables)

This paper contains 39 sections, 76 equations, 6 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Background
  4. From PPO to GPO
  5. Action Space Transformation
  6. Effect of action space Growth
  7. Update Equivalence.
  8. Gradient Difference Bound.
  9. Early-Stage Optimization: Signal-to-Noise Ratio and Local Convergence
  10. Gradient Variance.
  11. Signal-to-Noise Ratio.
  12. Convergence Error Bound.
  13. Early-Stage Return Advantage.
  14. Late-Stage Behavior: Exploration and Asymptotic Performance
  15. Experiments
  16. ...and 24 more sections

Figures (6)

  • Figure 1: Overview of GPO. GPO gradually expands the effective action space during training, enabling stable early optimization and improving asymptotic performance.
  • Figure 2: Simulated quadruped (left) and hexapod robot (right).
  • Figure 3: Training rewards under different growth functions. Left: quadruped whole-body. Right: hexapod locomotion.
  • Figure 4: Joint-level simulation results for the front-left leg. Top row: quadruped. Bottom row: hexapod. For each robot, columns from left to right show (1) joint torque with GPO, (2) joint torque with PPO, (3) joint velocity with GPO, and (4) joint velocity with PPO.
  • Figure 5: Hexapod gait and body height visualization. Left: GPO; right: PPO. For each method, the top shows six-leg contact patterns, and the bottom compares target and actual body height.
  • ...and 1 more figures