GUESS:GradUally Enriching SyntheSis for Text-Driven Human Motion Generation

TL;DR

GUESS introduces GradUally Enriching SyntheSis, a coarse-to-fine text-driven motion generation framework that progressively abstracts a human pose into multiple skeleton scales and employs a cascaded latent diffusion model with dynamic conditioning. Motion is encoded with per-scale VAEs to produce latent embeddings, which are then refined from a coarse $z_4$ to a fine $z_1$ through denoising steps guided by both textual input and evolving coarse-motion prompts. A dynamic multi-condition fusion mechanism adaptively weights textual and coarse-motion cues during diffusion, improving text-motion alignment and output diversity. Extensive experiments on multiple benchmarks show significant gains over state-of-the-art methods in fidelity, realism, and diversity for both text-to-motion and action-to-motion tasks, establishing GUESS as a strong baseline for cross-modal human motion synthesis.

Abstract

In this paper, we propose a novel cascaded diffusion-based generative framework for text-driven human motion synthesis, which exploits a strategy named GradUally Enriching SyntheSis (GUESS as its abbreviation). The strategy sets up generation objectives by grouping body joints of detailed skeletons in close semantic proximity together and then replacing each of such joint group with a single body-part node. Such an operation recursively abstracts a human pose to coarser and coarser skeletons at multiple granularity levels. With gradually increasing the abstraction level, human motion becomes more and more concise and stable, significantly benefiting the cross-modal motion synthesis task. The whole text-driven human motion synthesis problem is then divided into multiple abstraction levels and solved with a multi-stage generation framework with a cascaded latent diffusion model: an initial generator first generates the coarsest human motion guess from a given text description; then, a series of successive generators gradually enrich the motion details based on the textual description and the previous synthesized results. Notably, we further integrate GUESS with the proposed dynamic multi-condition fusion mechanism to dynamically balance the cooperative effects of the given textual condition and synthesized coarse motion prompt in different generation stages. Extensive experiments on large-scale datasets verify that GUESS outperforms existing state-of-the-art methods by large margins in terms of accuracy, realisticness, and diversity. Code is available at https://github.com/Xuehao-Gao/GUESS.

Figures (11)

• Figure 1: Given a textual description, the human brain imagines the corresponding motion visualization by inferring the body poses at the coarse body part level first and then enriching the finer motion details gradually.
• Figure 2: The Framework of GUESS. In the training phase, we first represent a human motion input with multiple pose scales ($\boldsymbol{S}_{1},\cdots,\boldsymbol{S}_{4}$) and train a motion embedding module to learn an effective latent motion representation ($z_{1},\cdots,z_{4}$) within each scale. Then, we train a cascaded latent-based diffusion model to learn a powerful probabilistic text-to-motion mapping with a joint guidance of gradually richer motion embedding ($z_{4}, \cdots, z_{2}$) and textual condition embedding ($c$). In the test phase, the motion inference module generates the motion embedding of the finest pose sale ($z_{1}$) from the text embedding and sends it to the corresponding decoder ($\mathcal{D}_{1}$) for the 3D motion reconstruction.
• Figure 3: Four body scales on HumanML3D dataset. In the initial scale $S_{1}$, each pose of HumanML3D skeleton contains 22 body-joint nodes. In $S_{2}$, $S_{3}$ and $S_{4}$, we consider 11, 5 and 1 body-part nodes, respectively.
• Figure 4: Iterative Denoising Module. Taking denoiser $\mathcal{R}_{3}$ as an example, its denoising process can be factorized into two sequential stages: Noise Prediction and Noise Elimination. Given the initial textual condition embedding $c$ and synthesized coarse human motion embedding $z_{4}$, $\mathcal{R}_{3}$ recursively infers the latent motion embedding $z_{3}$ from a sampled Gaussian noise signal $z_{T}$ with $T$ Markov denoising steps.
• Figure 5: Dynamic Multi-Condition Fusion Module. Taking the dynamic multi-condition fusion in $\mathcal{R}_{3}$ as an example, it adaptively infers the response score of $c$ and $z_{4}$ at $t$-th Markov denoising step. Based on the inferred channel-wise attention and cross-modal attention, $c$ and $z_{4}$ are integrated into a joint condition embedding effectively.