Advanced Skills through Multiple Adversarial Motion Priors in Reinforcement Learning

TL;DR

This work presents an approach to enhance the concept of adversarial motion prior-based RL, allowing for multiple, discretely switchable motion styles, and demonstrates that multiple styles and skills can be learned simultaneously without significant performance differences, even in combination with motion data-free skills.

Abstract

In recent years, reinforcement learning (RL) has shown outstanding performance for locomotion control of highly articulated robotic systems. Such approaches typically involve tedious reward function tuning to achieve the desired motion style. Imitation learning approaches such as adversarial motion priors aim to reduce this problem by encouraging a pre-defined motion style. In this work, we present an approach to augment the concept of adversarial motion prior-based RL to allow for multiple, discretely switchable styles. We show that multiple styles and skills can be learned simultaneously without notable performance differences, even in combination with motion data-free skills. Our approach is validated in several real-world experiments with a wheeled-legged quadruped robot showing skills learned from existing RL controllers and trajectory optimization, such as ducking and walking, and novel skills such as switching between a quadrupedal and humanoid configuration. For the latter skill, the robot is required to stand up, navigate on two wheels, and sit down. Instead of tuning the sit-down motion, we verify that a reverse playback of the stand-up movement helps the robot discover feasible sit-down behaviors and avoids tedious reward function tuning.

Figures (6)

• Figure 1: Quadruped-humanoid transformer ( https://youtu.be/kEdr0ARq48A) with a time-lapse from left to right of a stand-up and sit-down motion (top image), obstacle negotiation (middle image), and indoor navigation (bottom images). The former skill and the humanoid navigation on two legs are achieved through traditional RL training with a task reward formulation. Instead of tuning the sit-down skill, we can reverse the playback of the stand-up motion and use it as a motion prior that helps the robot discover feasible sit-down behaviors avoiding tedious reward function tuning.
• Figure 2: Multi-AMP overview: The discriminator predicts a style reward $s_t^{style}$ which is high if the policy's behavior is similar to the motions of the motion-data base $M^i$, by distinguishing between state transitions $(s_t, s_{t+1})$ of both sources. The style reward is added to the task reward, which finally leads to the policy fulfilling the task while applying the motion data's style.
• Figure 3: Four-legged locomotion (top row) and ducking motion (bottom row) of the motion data source (left column), simulation training (center column), and final deployment on the real robot using Multi-AMP. The former skill is trained with a motion prior from a different simulation environment and control approach, while the ducking motion is trained with data from trajectory optimization bjelonic2022complex.
• Figure 4: Stand up-sequence in simulation and on the real robot. The policy is able to stand up, navigate large distances on two legs, and finally sit down again using the stand-up motion prior.
• Figure 5: Comparison of the sitting down motions. Top row: If the agent learns to sit down with task rewards only, it falls forward with extended front legs, which causes high impacts and leads to over-torque on the real robot. Marked in blue is the trajectory of the center of gravity of the base. Bottom row: When sitting down with task reward and style reward from the reversed stand-up sequence, the robot squats down to lower its center of gravity before tilting forward, thereby reducing the impact's magnitude. Marked in green is the trajectory of the center of gravity of the base. We note that compared to the previous case the base is lowered in a way that causes less vertical base velocity at the moment of impact.