Physics-Based Motion Imitation with Adversarial Differential Discriminators

TL;DR

This work tackles manual reward engineering in multi-objective RL for physics-based motion imitation by introducing Adversarial Differential Discriminator (ADD), a GAN-based framework that uses a differential objective vector and a single positive sample to automatically balance objectives. ADD replaces linear reward aggregation with a nonlinear adversarial aggregator, enabling precise frame-level imitation and capturing complex objective interactions while adapting as training progresses. The approach yields motion-tracking performance comparable to state-of-the-art methods on diverse motions without handcrafted rewards and extends to non-imitation tasks, with gradient-penalty ablations highlighting the importance of regularization. Overall, ADD offers a general, data-efficient mechanism for multi-objective RL in animation and robotics, reducing manual tuning and improving robustness across morphologies and tasks.

Abstract

Multi-objective optimization problems, which require the simultaneous optimization of multiple objectives, are prevalent across numerous applications. Existing multi-objective optimization methods often rely on manually-tuned aggregation functions to formulate a joint optimization objective. The performance of such hand-tuned methods is heavily dependent on careful weight selection, a time-consuming and laborious process. These limitations also arise in the setting of reinforcement-learning-based motion tracking methods for physically simulated characters, where intricately crafted reward functions are typically used to achieve high-fidelity results. Such solutions not only require domain expertise and significant manual tuning, but also limit the applicability of the resulting reward function across diverse skills. To bridge this gap, we present a novel adversarial multi-objective optimization technique that is broadly applicable to a range of multi-objective reinforcement-learning tasks, including motion tracking. Our proposed Adversarial Differential Discriminator (ADD) receives a single positive sample, yet is still effective at guiding the optimization process. We demonstrate that our technique can enable characters to closely replicate a variety of acrobatic and agile behaviors, achieving comparable quality to state-of-the-art motion-tracking methods, without relying on manually-designed reward functions. Code and results are available at https://add-moo.github.io/.

Figures (15)

• Figure 1: Snapshots of the simulated humanoid characters trained using ADD performing various skills. ADD enables characters to replicate a diverse repertoire of behaviors, achieving tracking quality comparable to state-of-the-art motion imitation methods, without manual reward engineering.
• Figure 2: Visual snapshots of the simulated EVAL robot replicating a range of target motions, using motion-tracking controllers trained with ADD. The controllers, trained using learned tracking rewards, successfully enable the robot to reproduce a set of challenging skills.
• Figure 3: Learning curves of AMP, DeepMimic, and ADD, each trained with 5 different random seeds. ADD demonstrates better consistency than DeepMimic across different seeds. DeepMimic policies converge to suboptimal behaviors in half of the seeds when tracking the Backflip and Cartwheel motions.
• Figure 4: Qualitative results of ADD on the Walker task, a standard RL benchmark. The walker trained using ADD exhibits behaviors comparable in quality to those learned with manually designed reward functions from 2018DMControl.
• Figure 5: Qualitative results of ADD on training a UniTree Go1 quadruped to walk. The arrow denotes the steering command. Compared to controllers trained with manually tuned rewards legged-gym-pmlr-v164-rudin22a, the ADD Go1 policy displays more natural gaits, with greater foot lift and longer strides.