Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrases

TL;DR

The paper tackles the fundamental gap between human motion and action semantics, reframing the problem as a many-to-many mapping and proposing Kinematic Phrases (KP) as an objective, interpretable intermediate representation of motion. KP distills motion into six types totaling 392 phrases, enabling a unified motion knowledge base and a white-box benchmarking framework called Kinematic Prompt Generation (KPG) for text-to-motion evaluation. By learning a joint motion–KP latent space with dual VAEs and alignment losses, the approach supports interpolation, modification, and generation while promoting robustness through partial KP corruption. Through extensive experiments and user studies, KP-based methods demonstrate improved interpretability, controllability, and new insights into semantic consistency, while exposing limitations of traditional metrics and highlighting the potential for reliable, white-box evaluation in motion synthesis.

Abstract

Motion understanding aims to establish a reliable mapping between motion and action semantics, while it is a challenging many-to-many problem. An abstract action semantic (i.e., walk forwards) could be conveyed by perceptually diverse motions (walking with arms up or swinging). In contrast, a motion could carry different semantics w.r.t. its context and intention. This makes an elegant mapping between them difficult. Previous attempts adopted direct-mapping paradigms with limited reliability. Also, current automatic metrics fail to provide reliable assessments of the consistency between motions and action semantics. We identify the source of these problems as the significant gap between the two modalities. To alleviate this gap, we propose Kinematic Phrases (KP) that take the objective kinematic facts of human motion with proper abstraction, interpretability, and generality. Based on KP, we can unify a motion knowledge base and build a motion understanding system. Meanwhile, KP can be automatically converted from motions to text descriptions with no subjective bias, inspiring Kinematic Prompt Generation (KPG) as a novel white-box motion generation benchmark. In extensive experiments, our approach shows superiority over other methods. Our project is available at https://foruck.github.io/KP/.

Figures (13)

• Figure 1: The huge gap between motion and action semantics results in the many-to-many problem. We propose Kinematic Phrases (KP) as an intermediate to bridge the gap. KPs objectively capture human kinematic cues. It properly abstracts diverse motions with interpretability. As shown, the Phrases in the yellow box could capture key patterns of walk for diverse motions.
• Figure 2: HumanML3D hml3d metric values are approaching even surpassing GT-level while being increasingly indistinguishable. The higher-than-GT R-Precision could be insufficient as a semantic consistency indicator.
• Figure 3: Six types of KP from four kinematic hierarchies are extracted from a motion sequence. A scalar indicator $s_i$ is calculated per Phrase per frame. Its sign categorizes the corresponding Phrase.
• Figure 4: We train a motion-KP joint latent space to exploit the clarity and interpretability of KP. The space is then applied to multiple tasks, including motion interpolation, modification, and generation.
• Figure 5: Kinematic Prompt Generation. 7,796 prompts are converted from KP with templates, corresponding to certain KP patterns. We calculate the generation accuracy by checking the appearance of the patterns in KP extracted from the generated motion.