Diffusion-based Human Motion Style Transfer with Semantic Guidance

TL;DR

This work tackles unseen 3D human motion style transfer by decoupling generation and style translation through a two-stage diffusion framework. Stage I pre-trains a diffusion-based text-to-motion model as a robust generative prior and a motion-semantic discriminator, while Stage II fine-tunes with a single style example to perform style transfer via the reverse diffusion, guided by a style-example reconstruction loss in CLIP space and a semantic loss. The approach achieves state-of-the-art performance on unseen styles, maintaining content fidelity while embedding stylistic characteristics, and can even transfer style from video demonstrations, with practical implications for animation, games, and VR. The framework efficiently leverages unbalanced large-scale text-motion data for priors and enables few-shot adaptation, offering a scalable path toward rich, controllable motion synthesis and transfer.

Abstract

3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaIN- based methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN-based methods. Therefore, in this paper, we propose a novel two-stage framework for few-shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre-train a diffusion-based text-to-motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine-tune the pre-trained diffusion model in a few-shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion-style translation process since the motion styles can be viewed as special motion variations. During the fine-tuning for style transfer, a simple yet effective semantic-guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state-of-the-art performance and has practical applications.

Figures (8)

• Figure 1: The concept figure of our framework. (a) The pre-trained text-to-motion generative model; (b) The motion-to-motion translation model based on diffusion; (c) The fine-tuning for human motion style transfer.
• Figure 2: Overview of our learning process. In the pre-training stage, we train the diffusion-based T2M model $\varepsilon_{\theta}$ (a) as generative prior and a motion-semantic discriminator (b) for measuring the similarity between motion and textual prompts. In the fine-tuning stage, we employ the style example reconstruction loss (c) to utilize the style information provided by example $m^s$. Concurrently, we use the pre-trained motion-semantic discriminator to guide the transferred motion to match the semantics of the original content and the style of the example motion (d). Notably, input style example, generated neutral motion, content motion, and output transferred motion are colored light blue, grey, green, and yellow, respectively.
• Figure 3: The architecture of our motion-semantic discriminator. We introduce a trainable "token" (short for "semantic token") for motion pooling and align the motion-semantic feature $H_{token}$ with the text feature $H_{text}$ in the CLIP space.
• Figure 4: Qualitative results of our few-shot style transfer. We can transfer unseen motion styles (from Xia dataset) to different content motions (from HumanML3D test set).
• Figure 5: Qualitative results. We can learn the human motion style from video and transfer it to different content motions.