Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions

TL;DR

This work tackles the challenge of under-specified rewards in high-dimensional legged robotics by replacing hand-crafted, task-specific rewards with a learned style reward derived from motion-capture data via Adversarial Motion Priors (AMP). The style prior is trained through adversarial imitation against a dataset (as little as 4.5 seconds) and combined with a task reward to yield policies that resemble natural locomotion while achieving desired tasks. In simulation and on a real quadruped, AMP-enabled policies show energy-efficient gait transitions and robust transfer, outperforming hand-designed style rewards and avoiding the violent dynamics seen with no-style baselines. The approach reduces the need for labor-intensive reward engineering and offers a practical path toward naturalistic, transferable locomotion in real robots.

Abstract

Training a high-dimensional simulated agent with an under-specified reward function often leads the agent to learn physically infeasible strategies that are ineffective when deployed in the real world. To mitigate these unnatural behaviors, reinforcement learning practitioners often utilize complex reward functions that encourage physically plausible behaviors. However, a tedious labor-intensive tuning process is often required to create hand-designed rewards which might not easily generalize across platforms and tasks. We propose substituting complex reward functions with "style rewards" learned from a dataset of motion capture demonstrations. A learned style reward can be combined with an arbitrary task reward to train policies that perform tasks using naturalistic strategies. These natural strategies can also facilitate transfer to the real world. We build upon Adversarial Motion Priors -- an approach from the computer graphics domain that encodes a style reward from a dataset of reference motions -- to demonstrate that an adversarial approach to training policies can produce behaviors that transfer to a real quadrupedal robot without requiring complex reward functions. We also demonstrate that an effective style reward can be learned from a few seconds of motion capture data gathered from a German Shepherd and leads to energy-efficient locomotion strategies with natural gait transitions.

Figures (6)

• Figure 1: Training with Adversarial Motion Priors encourages the policy to produce behaviors which capture the essence of the motion capture dataset while satisfying the auxiliary task objective. Only a small amount of motion capture data is required to train the learning system (4.5 seconds in our experiments).
• Figure 2: Key frames, gait pattern, velocity tracking, and energy-efficiency of the robot dog throughout a trajectory A: Key frames of A1 during a canter motion overlaid on a plain background for contrast. B: Gait diagram indicating contact timing and duration for each foot in black. Training with Adversarial Motion Priors enables the policy to synthesize behaviors which lead to natural gait transitions at different velocities. C: Plot of commanded forward velocities and estimated velocities during the rollout. D: Estimated Cost of Transport (COT) during the rollout. While pacing the COT remains constant with small oscillations. However, when the robot enters a canter phase the COT exhibits spikes corresponding to the robot pushing off its hind legs and troughs corresponding to the flight phase where energy consumption is low. This gait transition phenomenon closely relates to the behavior of quadrupedal mammals, which modulate their gait according to their speed of travel, leading to minimal energy consumption consumption Hoyt1981.
• Figure 3: An agent trained with Adversarial Motion Priors extracts the naturalistic locomotion strategies found in the dataset and can change its gait based on the desired velocity. Top: when commanded to move at a low forward velocity $\left(0.8m\per s\right)$, the agent select a pacing gait. Bottom: when the commanded forward velocity increases to $\left(1.7m\per s\right)$, the agent switches to a trotting gait. The green low-opacity overlaid images show the previous frame for reference.
• Figure 4: By using Adversarial Motion Priors, the policy can deviate from the reference motion data to satisfy the desired velocity commands and navigate carefully through a route with sharp turns.
• Figure 5: The policy trained with no style reward learns to exploit inaccurate simulator dynamics and violently vibrates the simulated robot's feet on the ground to move. The high motor velocities and torques make it impossible to deploy this control strategy on the real robot.