D-LORD for Motion Stylization

TL;DR

D-LORD tackles motion stylization by disentangling motion into class, content, and aleatoric uncertainty latent spaces through a two-stage latent optimization and amortization framework. It avoids paired data, employs AdaIN for style transfer, and uses a VAE to map the aleatoric latent to a Gaussian distribution to enable diverse motion generation, while a skeleton-consistency loss stabilizes bone lengths. The approach achieves superior style transfer and content preservation on CMU Xia, MHAD, and RRIS, and supports robust motion retargeting with the ability to adjust gait features across identities. This non-adversarial, generalizable framework advances motion synthesis and retargeting in animation and robotics, offering a scalable path to diverse, label-conditioned motion generation.

Abstract

This paper introduces a novel framework named D-LORD (Double Latent Optimization for Representation Disentanglement), which is designed for motion stylization (motion style transfer and motion retargeting). The primary objective of this framework is to separate the class and content information from a given motion sequence using a data-driven latent optimization approach. Here, class refers to person-specific style, such as a particular emotion or an individual's identity, while content relates to the style-agnostic aspect of an action, such as walking or jumping, as universally understood concepts. The key advantage of D-LORD is its ability to perform style transfer without needing paired motion data. Instead, it utilizes class and content labels during the latent optimization process. By disentangling the representation, the framework enables the transformation of one motion sequences style to another's style using Adaptive Instance Normalization. The proposed D-LORD framework is designed with a focus on generalization, allowing it to handle different class and content labels for various applications. Additionally, it can generate diverse motion sequences when specific class and content labels are provided. The framework's efficacy is demonstrated through experimentation on three datasets: the CMU XIA dataset for motion style transfer, the MHAD dataset, and the RRIS Ability dataset for motion retargeting. Notably, this paper presents the first generalized framework for motion style transfer and motion retargeting, showcasing its potential contributions in this area.

Figures (15)

• Figure 1: D-LORD framework for Motion stylization.
• Figure 2: Stage 1: Latent optimization. Parameters of the generator and all the class, content, and AU embeddings are optimized using SGD optimizer. All motion sequences of the same class share a single class embedding and those of the same content share a single content embedding. The network is trained using Equation \ref{['eq_4']}. Once the network is trained, the latent spaces of the training set are disentangled.
• Figure 3: Stage 2: (a) Amortization. All three encoders are trained to generate the optimized embeddings of stage 1 for a given motion sequence. The model is trained using Equation \ref{['eq_6']}. (b) Variational autoencoder that is used to map AU latents to another latent space that follows the Gaussian distribution.
• Figure 4: Motion stylization / Diverse motion sequence generation / Inference: Diverse motion sequences from a given motion sequence can be generated by sampling multiple AU latents from Gaussian distribution. During inference, the motion sequence whose style is to be transferred is given as input to the class encoder, and the motion sequence whose motion is to be stylized is given as input to the content encoder.
• Figure 5: Residual Block