KETA: Kinematic-Phrases-Enhanced Text-to-Motion Generation via Fine-grained Alignment

TL;DR

KETA tackles the mismatch between linguistic descriptions and physical motion by introducing fine-grained alignment between decomposed text and kinematic phrases (KP). By decomposing prompts with a language model and aligning to KP segments, KETA constrains diffusion-based motion generation to be more physically consistent. During training, an auxiliary text-KP alignment loss guides both encoder and KP projection; during inference, iterative refinement uses KP-text distance as guidance. Empirical results show substantial improvements in R-Precision and FID over base diffusion backbones and several state-of-the-art T2M models, demonstrating the value of physical-state-aware, KP-driven alignment for motion generation.

Abstract

Motion synthesis plays a vital role in various fields of artificial intelligence. Among the various conditions of motion generation, text can describe motion details elaborately and is easy to acquire, making text-to-motion(T2M) generation important. State-of-the-art T2M techniques mainly leverage diffusion models to generate motions with text prompts as guidance, tackling the many-to-many nature of T2M tasks. However, existing T2M approaches face challenges, given the gap between the natural language domain and the physical domain, making it difficult to generate motions fully consistent with the texts. We leverage kinematic phrases(KP), an intermediate representation that bridges these two modalities, to solve this. Our proposed method, KETA, decomposes the given text into several decomposed texts via a language model. It trains an aligner to align decomposed texts with the KP segments extracted from the generated motions. Thus, it's possible to restrict the behaviors for diffusion-based T2M models. During the training stage, we deploy the text-KP alignment loss as an auxiliary goal to supervise the models. During the inference stage, we refine our generated motions for multiple rounds in our decoder structure, where we compute the text-KP distance as the guidance signal in each new round. Experiments demonstrate that KETA achieves up to 1.19x, 2.34x better R precision and FID value on both backbones of the base model, motion diffusion model. Compared to a wide range of T2M generation models. KETA achieves either the best or the second-best performance.

Figures (5)

• Figure 1: Overview of KETA, a physical-state-aware T2M model via fine-grained alignment between text and KP.
• Figure 2: Left: The Motion Diffusion Model (MDM) overview. The input is the motion sequence of length, the current timestep $t$, and a conditioning code, then a Text Encoder projects and adds them to get the input token $z_{tk}$. During each step, the clean motion is predicted. Right: MDM Sampling. Given a condition $c$, the model iterates to get the refined motion predicting a clean sample and diffuses it back in each round.
• Figure 3: The text decomposition process and the alignment model structure.
• Figure 4: Comparison between KETA and MDM, both the transformer encoder and decoder structure of KETA outperform their MDM counterparts in both FID and R-precision.
• Figure 5: The motions generated by KETA compared with MDM, our generated motions are consistent with the text prompts, showing higher text-to-motion similarity. Meanwhile, MDM fails to achieve spatial and temporal consistency.