CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion

TL;DR

Quantitative simulation comparisons show that the proposed Concurrent Teacher-Student reinforcement learning architecture for legged locomotion over uneven terrains reduces the average velocity tracking error by up to 20% compared to the two-stage teacher-student, demonstrating significant superiority in addressing blind locomotion tasks.

Abstract

Thanks to recent explosive developments of data-driven learning methodologies, reinforcement learning (RL) emerges as a promising solution to address the legged locomotion problem in robotics. In this paper, we propose CTS, a novel Concurrent Teacher-Student reinforcement learning architecture for legged locomotion over uneven terrains. Different from conventional teacher-student architecture that trains the teacher policy via RL first and then transfers the knowledge to the student policy through supervised learning, our proposed architecture trains teacher and student policy networks concurrently under the reinforcement learning paradigm. To this end, we develop a new training scheme based on a modified proximal policy gradient (PPO) method that exploits data samples collected from the interactions between both the teacher and the student policies with the environment. The effectiveness of the proposed architecture and the new training scheme is demonstrated through substantial quantitative simulation comparisons with the state-of-the-art approaches and extensive indoor and outdoor experiments with quadrupedal and point-foot bipedal robot platforms, showcasing robust and agile locomotion capability. Quantitative simulation comparisons show that our approach reduces the average velocity tracking error by up to 20% compared to the two-stage teacher-student, demonstrating significant superiority in addressing blind locomotion tasks. Videos are available at https://clearlab-sustech.github.io/concurrentTS.

Figures (9)

• Figure 1: CTS enables legged robots of various sizes and configurations to achieve robust and agile locomotion across challenging real-world terrains, while also possessing exceptional capabilities to withstand strong external disturbances.
• Figure 2: Overview of the learning framework. The teacher and student policies are trained concurrently using PPO within an asymmetric actor-critic framework. Agents in both groups share the same critic and policy network, with actions determined by observations and latent representations from either privileged or proprioceptive encoder. The privileged encoder is trained via policy gradient, while the proprioceptive encoder undergoes supervised learning to minimize reconstruction loss.
• Figure 3: Terrains in simulation.
• Figure 4: Learning curves of average terrain level.
• Figure 5: Evaluation of average tracking error in four types of terrains. Linear velocity commands were uniformly sampled from $[-1.0, 1.0]$ m/s.

Theorems & Definitions (1)

• Remark 1