Towards High-Quality 3D Motion Transfer with Realistic Apparel Animation

TL;DR

The paper tackles high-quality 3D motion transfer for stylized characters with realistic apparel by introducing a data-driven, modular pipeline that disentangles body and apparel deformation. It leverages a new MMDMC dataset with detailed apparel annotations to train apparel segmentation, body skinning with geodesic priors, and non-linear apparel deformation conditioned on historical states, followed by a joint refinement stage. Quantitative and qualitative results on MMDMC (and qualitative Mixamo evaluations) demonstrate superior apparel realism and reduced body-apparel penetration compared with baselines, with ablations confirming the value of each component. The work enables more believable digital avatars in film and games and provides a publicly released dataset to foster further research in apparel-aware motion transfer.

Abstract

Animating stylized characters to match a reference motion sequence is a highly demanded task in film and gaming industries. Existing methods mostly focus on rigid deformations of characters' body, neglecting local deformations on the apparel driven by physical dynamics. They deform apparel the same way as the body, leading to results with limited details and unrealistic artifacts, e.g. body-apparel penetration. In contrast, we present a novel method aiming for high-quality motion transfer with realistic apparel animation. As existing datasets lack annotations necessary for generating realistic apparel animations, we build a new dataset named MMDMC, which combines stylized characters from the MikuMikuDance community with real-world Motion Capture data. We then propose a data-driven pipeline that learns to disentangle body and apparel deformations via two neural deformation modules. For body parts, we propose a geodesic attention block to effectively incorporate semantic priors into skeletal body deformation to tackle complex body shapes for stylized characters. Since apparel motion can significantly deviate from respective body joints, we propose to model apparel deformation in a non-linear vertex displacement field conditioned on its historic states. Extensive experiments show that our method produces results with superior quality for various types of apparel. Our dataset is released in https://github.com/rongakowang/MMDMC.

Figures (7)

• Figure 1: We present a novel method which transfers a source motion onto a target stylized character and generates realistic apparel animation.
• Figure 2: Illustration of our method. Given an input character (a), we aim to animate it following a reference 3D motion (b) and produce the target result (c). Previous methods mostly predict the skinning weights (d) with respect to body joints and deform the entire character via the LBS method (e), essentially treating the apparel equally with the body. Such approach lacks visual details and often contains unrealistic artifacts, such as body-apparel penetration. In contrast, we propose a novel pipeline that discriminates apparel vertices (in red) by apparel segmentation (f) and then explicitly models its local deformation, thus producing realistic apparel animation (g).
• Figure 3: Comparison of rig annotations with existing datasets. Existing datasets RigNetMixamo mostly provide rigging on body parts, while the apparel is not rigged in detail, hence the apparel can not independently deform. In contrast, our dataset contains dense apparel rigs, thus enabling realistic ground truth apparel animation.
• Figure 4: Overview of our method. Given the input character $\mathbf{V}$ (with known joint positions), our model first discriminates body ($\mathbf{B}$, blue) and apparel ($\mathbf{A}$, red) vertices in an apparel segmentation module. With the reference joints motion $\mathbf{T}^{(t)}$, we propose a geodesic attention block to estimate the skinning weight $\mathbf{W}$ and deform the body via the LBS method. Moreover, we model non-linear apparel displacement conditioned on historic states and joint motions. Finally, we jointly refine outputs from both modules to obtain the overall result $\hat{\mathbf{V}}^{(t)}$.
• Figure 5: Qualitative comparison. Our method produce superior results than baseline methods li2021learningliao2022skeleton that both contain artifacts on body or apparel (in circles). Moreover, we generate more realistic apparel results as highlighted in red at the right (using the GT apparel mask for consistent visualization of baseline methods).