Progressive Human Motion Generation Based on Text and Few Motion Frames

TL;DR

This paper introduces Text-Frame-to-Motion (TF2M), a controllable motion generation task that combines natural language descriptions with a few given frames to specify desired postures. It proposes Progressive Motion Generation (PMG), a diffusion-based framework that generates motion in multiple stages, progressively conditioning on text, given frames, and previously generated frames, guided by a Frame-aware Semantics Decoder and Text-Frame Guided Blocks. A Pseudo-frame Replacement Strategy during training mitigates train-test discrepancies from multi-stage frame accumulation, and a MotionCLIP evaluator complements traditional metrics to better assess text-motion alignment. Experiments on HumanML3Dt2m and KIT-MLkit show PMG achieves state-of-the-art performance, with strong robustness to frame quality and improved FID and alignment metrics, highlighting the potential for practical frame-guided motion control in AR/VR and animation pipelines.

Abstract

Although existing text-to-motion (T2M) methods can produce realistic human motion from text description, it is still difficult to align the generated motion with the desired postures since using text alone is insufficient for precisely describing diverse postures. To achieve more controllable generation, an intuitive way is to allow the user to input a few motion frames describing precise desired postures. Thus, we explore a new Text-Frame-to-Motion (TF2M) generation task that aims to generate motions from text and very few given frames. Intuitively, the closer a frame is to a given frame, the lower the uncertainty of this frame is when conditioned on this given frame. Hence, we propose a novel Progressive Motion Generation (PMG) method to progressively generate a motion from the frames with low uncertainty to those with high uncertainty in multiple stages. During each stage, new frames are generated by a Text-Frame Guided Generator conditioned on frame-aware semantics of the text, given frames, and frames generated in previous stages. Additionally, to alleviate the train-test gap caused by multi-stage accumulation of incorrectly generated frames during testing, we propose a Pseudo-frame Replacement Strategy for training. Experimental results show that our PMG outperforms existing T2M generation methods by a large margin with even one given frame, validating the effectiveness of our PMG. Code is available at https://github.com/qinghuannn/PMG.

Figures (11)

• Figure 1: Illustration of the Text-Frame-to-Motion generation task (a) and our proposed method (b). This task focuses on generating motions from text descriptions and very few given frames. The given frames are in yellow.
• Figure 2: Overview of our Progressive Motion Generation (PMG). (a) Our PMG is a diffusion-based model and generates the target motion in $K$ stages, as shown in \ref{['fig:intro']}. During the $k$-th stage, frames $M^k$ are generated via the denoising process, where $\mathbf{x}^k_T \sim \mathcal{N}(\textbf{0},\textbf{I})$. To generate $\mathbf{x}^k_{t-1}$ conditioned on the text, given frames, and frames generated in previous stages, we propose a Text-Frame Guided Generator, which mainly contains a Frame-aware Semantics Decoder and $L_2$ Text-Frame Guided Blocks. Note that $M^0$ denotes the given frames, and the pretrained language model is kept fixed. (b) Illustration of Text-Frame Guided Block. (c) Illustration of Fusion Module. $N_m$ denotes the number of generating frames.
• Figure 3: The illustration of our MotionCLIP, which maps matched text-motion pairs into closely aligned feature vectors in the latent space. MotionCLIP consists of two main components: a motion encoder and a text encoder.
• Figure 4: Visual comparisons on HumanML3D t2m dataset. The test sample is #013150. We visualize the generated motion of PMG when given one frame (indicated in yellow). The motion generated by PMG is more consistent with the given text.
• Figure 5: Visual comparisons on HumanML3D t2m dataset. The test samples are #000534 and #010797. Note that these two samples are hard samples. We visualize the generated motions of our PMG when given different frames (indicated in yellow).