Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response

TL;DR

<3-5 sentence high-level summary>Hybrid Internal Model (HIM) introduces an Internal Model Control–inspired disturbance estimator that uses the robot’s own response to implicitly infer external states such as terrain friction and elevation. It learns a hybrid internal embedding consisting of an explicit velocity estimate and an implicit stability-related latent, optimized via contrastive learning (SwAV-style) and integrated with a PPO policy trained using proprioception-only sensors. The approach leverages massively parallel simulation in Isaac Gym with dynamics randomization and a terrain curriculum to achieve robust sim2real transfer, demonstrated on multiple quadruped platforms across real-world tasks and unseen terrains. HIM delivers strong open-world generalization, excellent sample efficiency, and requires minimal sensing and training time (approximately 1 hour on an RTX 4090).

Abstract

Robust locomotion control depends on accurate state estimations. However, the sensors of most legged robots can only provide partial and noisy observations, making the estimation particularly challenging, especially for external states like terrain frictions and elevation maps. Inspired by the classical Internal Model Control principle, we consider these external states as disturbances and introduce Hybrid Internal Model (HIM) to estimate them according to the response of the robot. The response, which we refer to as the hybrid internal embedding, contains the robot's explicit velocity and implicit stability representation, corresponding to two primary goals for locomotion tasks: explicitly tracking velocity and implicitly maintaining stability. We use contrastive learning to optimize the embedding to be close to the robot's successor state, in which the response is naturally embedded. HIM has several appealing benefits: It only needs the robot's proprioceptions, i.e., those from joint encoders and IMU as observations. It innovatively maintains consistent observations between simulation reference and reality that avoids information loss in mimicking learning. It exploits batch-level information that is more robust to noises and keeps better sample efficiency. It only requires 1 hour of training on an RTX 4090 to enable a quadruped robot to traverse any terrain under any disturbances. A wealth of real-world experiments demonstrates its agility, even in high-difficulty tasks and cases never occurred during the training process, revealing remarkable open-world generalizability.

Figures (8)

• Figure 1: Our locomotion policy can drive robots to walk across any terrain under any disturbances. Key insight lies in alternatively estimating environmental dynamics with the response of the robot.
• Figure 2: Overview of our framework. The policy network receives partial observations and the hybrid internal embedding, which is optimized to the robot's successor state with contrastive learning. The framework is alternatively optimized with HIO and PPO.
• Figure 3: (a) IMC and (b) our implementations.
• Figure 4: Latent space visualizations of (a) our hybrid internal model and (b) nahrendra2023dreamwaq.
• Figure 5: Ablation studies with learning curves of (a) normalized linear velocity tracking score, (b) normalized angular velocity tracking score, and (c) maximum reachable terrain level in Isaac Gym.