Scaling Rough Terrain Locomotion with Automatic Curriculum Reinforcement Learning

TL;DR

This paper tackles the challenge of scaling reinforcement learning for legged locomotion in unstructured, multi-axis task spaces by introducing LP-ACRL, a curriculum-learning framework that updates task sampling online using learning-progress signals derived from episodic rewards. LP-ACRL removes the need for hand-designed curricula and demonstrates strong scalability across flat and rough terrains, including large-scale task spaces with hundreds of task instances. The approach yields faster convergence, higher final competence, and successful transfer to the real ANYmal D platform via teacher–student distillation, achieving high-speed, robust locomotion under diverse conditions. The results establish learning-progress-driven task selection as an effective principle for automatic curriculum generation in complex robotic learning settings, with practical implications for real-world deployment and future research in scalable curriculum RL.

Abstract

Curriculum learning has demonstrated substantial effectiveness in robot learning. However, it still faces limitations when scaling to complex, wide-ranging task spaces. Such task spaces often lack a well-defined difficulty structure, making the difficulty ordering required by previous methods challenging to define. We propose a Learning Progress-based Automatic Curriculum Reinforcement Learning (LP-ACRL) framework, which estimates the agent's learning progress online and adaptively adjusts the task-sampling distribution, thereby enabling automatic curriculum generation without prior knowledge of the difficulty distribution over the task space. Policies trained with LP-ACRL enable the ANYmal D quadruped to achieve and maintain stable, high-speed locomotion at 2.5 m/s linear velocity and 3.0 rad/s angular velocity across diverse terrains, including stairs, slopes, gravel, and low-friction flat surfaces--whereas previous methods have generally been limited to high speeds on flat terrain or low speeds on complex terrain. Experimental results demonstrate that LP-ACRL exhibits strong scalability and real-world applicability, providing a robust baseline for future research on curriculum generation in complex, wide-ranging robotic learning task spaces.

Figures (9)

• Figure 1: Real World Deployment. Policy trained with LP-ACRL enables the ANYmal D robot to traverse diverse rough terrains, achieving speeds up to 2.5 m/s and 3.0 rad/s. Supplementary videos and implementation details are available on our project webpage https://sites.google.com/view/lp-acrl.
• Figure 2: Learning Progress-based Automatic Curriculum Reinforcement Learning.The task-sampling distribution is updated based on learning progress, prioritizing policy learning on the most informative tasks for further improvement.
• Figure 3: Episodic Percentage Tracking Error with Stability Penalty (EPTE-SP, $\tilde{\varepsilon}$) evaluation. The horizontal axis represents task instances, defined by the range of the absolute commanded linear velocity (in m/s). Bars indicate the mean performance averaged over multiple random seeds, with error bars representing the minimum and maximum values. The results demonstrate that the policy trained via LP-ACRL significantly outperforms all baseline policies,consistently maintaining the lowest EPTE-SP and narrowest variability across most intervals, while preserving its superiority in mean performance even in the most challenging task instance.
• Figure 4: Evolution of Episodic Reward throughout the training process. Subplots represent distinct task instances, defined by the range of the absolute commanded linear velocity (in m/s). The horizontal axis denotes training iterations. Solid curves indicate the mean reward averaged over multiple random seeds, with shaded regions indicating the variability. The results demonstrate that the policy trained via LP-ACRL exhibits significantly faster convergence and superior asymptotic performance compared to baselines across the task instances.
• Figure 5: Task-sampling distribution under different curriculum methods. The vertical axis denotes the eight task instances (indices 0-7), corresponding to the eight velocity tracking ranges. The colormap indicates the sampling probability of each task instance. LP-ACRL rapidly infers the underlying difficulty structure early in training, subsequently modulating sampling to balance high-difficulty exploration with the retention of mastered tasks.