Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model Homotopy Transfer

TL;DR

Dynamic motions for legged robots are impeded by the gap between simplified models and full-body dynamics. The paper introduces a continuation-based framework that pretrains a policy on a Single Rigid Body (SRB) model and progressively transfers it to full-body dynamics via model homotopy, using a continuation parameter $\lambda$ to redistribute mass and inertia. Empirical results show faster convergence, improved stability, and enhanced disturbance robustness across gaits, flips, and wall-assisted maneuvers, with successful real-world deployment on a quadrupedal robot. This approach provides a practical pathway for sim-to-real transfer and dynamic behavior discovery in legged robotics, reducing reliance on reward engineering and demonstrations. It also highlights the potential of model-continuum strategies to bridge ROM-based learning and high-fidelity control in complex contact-rich environments.

Abstract

Generating dynamic motions for legged robots remains a challenging problem. While reinforcement learning has achieved notable success in various legged locomotion tasks, producing highly dynamic behaviors often requires extensive reward tuning or high-quality demonstrations. Leveraging reduced-order models can help mitigate these challenges. However, the model discrepancy poses a significant challenge when transferring policies to full-body dynamics environments. In this work, we introduce a continuation-based learning framework that combines simplified model pretraining and model homotopy transfer to efficiently generate and refine complex dynamic behaviors. First, we pretrain the policy using a single rigid body model to capture core motion patterns in a simplified environment. Next, we employ a continuation strategy to progressively transfer the policy to the full-body environment, minimizing performance loss. To define the continuation path, we introduce a model homotopy from the single rigid body model to the full-body model by gradually redistributing mass and inertia between the trunk and legs. The proposed method not only achieves faster convergence but also demonstrates superior stability during the transfer process compared to baseline methods. Our framework is validated on a range of dynamic tasks, including flips and wall-assisted maneuvers, and is successfully deployed on a real quadrupedal robot.

Figures (8)

• Figure 1: Conceptual illustration of the proposed model homotopy transfer. The framework defines a continuation path from the simplified SRB to full-body dynamics via gradual mass and inertia redistribution. This approach facilitates the stable transfer of core motion patterns to complex, highly dynamic behaviors such as wall-assisted maneuvers.
• Figure 2: Overview of the proposed learning framework. The motion policy is first pretrained on a simplified SRB model to learn core motion patterns. Subsequently, this policy is smoothly adapted to the full-body environment using our model homotopy transfer.
• Figure 3: Snapshots of wall-assisted backflip. The red lines represent the CoM trajectories over time.
• Figure 4: Comparison of training performance across various methods and motions. (a) Converged iteration. The number of iterations required for the return to reach 99% of the final mean value. (b) Normalized return. The black box on top of each bar indicates the range between the maximum and minimum normalized returns. (c) Learning curves for wall-assisted backflip. The solid line represents the average, while the shaded region shows the range between maximum and minimum values. Overall, the proposed Model Homotopy Transfer demonstrated the fastest convergence and the most consistent performance.
• Figure 5: Comparison of robustness across disturbances. Each heatmap shows the success rate for combinations of force and torque disturbance norms, with brighter colors indicating higher success rates.