The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion

TL;DR

The paper tackles unifying verbal and non-verbal language of 3D human motion by building a multimodal language model that can ingest text, speech, or motion as input and generate either motions or text. It introduces a compositional motion space encoded by four-part tokenization and leverages HuBERT audio tokens and SentencePiece text tokens within a shared multimodal vocabulary. A two-stage training pipeline—modality alignment pre-training and instruction-following post-training—achieves state-of-the-art co-speech gesture generation with data-efficient learning, and enables novel tasks such as editable gesture generation and emotion prediction from motion, all grounded in a unified SMPL-X based motion representation. The framework promises practical impact for realistic virtual characters in games, films, and VR by enabling flexible, instruction-driven cross-modal motion generation across modalities.

Abstract

Human communication is inherently multimodal, involving a combination of verbal and non-verbal cues such as speech, facial expressions, and body gestures. Modeling these behaviors is essential for understanding human interaction and for creating virtual characters that can communicate naturally in applications like games, films, and virtual reality. However, existing motion generation models are typically limited to specific input modalities -- either speech, text, or motion data -- and cannot fully leverage the diversity of available data. In this paper, we propose a novel framework that unifies verbal and non-verbal language using multimodal language models for human motion understanding and generation. This model is flexible in taking text, speech, and motion or any combination of them as input. Coupled with our novel pre-training strategy, our model not only achieves state-of-the-art performance on co-speech gesture generation but also requires much less data for training. Our model also unlocks an array of novel tasks such as editable gesture generation and emotion prediction from motion. We believe unifying the verbal and non-verbal language of human motion is essential for real-world applications, and language models offer a powerful approach to achieving this goal. Project page: languageofmotion.github.io.

Figures (9)

• Figure 1: We introduce a language-model-based motion understanding and generation framework that takes in any of the audio/motion/text modalities and outputs the desired target modality. Coupled with our generative pre-training strategy, our model demonstrates competitive performance on an array of tasks, showing promising signs toward unified verbal and non-verbal language of human motions.
• Figure 2: Method overview. We employ modality-specific tokenizers to process various input modalities. Specifically, we train a compositional body motion VQ-VAE to tokenize face, hands, upper body, and lower body motions into discrete tokens, combining these modality-specific vocabularies(audio and text) into a unified multimodal vocabulary. During training, mixed tokens from different modalities are used as input, and the output is generated through an encoder-decoder language model. The mixed tokens are fed into the transformer encoder, while the decoder predicts the probability distribution of the next token in an autoregressive manner at each step.
• Figure 3: Illustration of pre-training. We pre-train our language model by translating one modality to another using paired data.
• Figure 4: Qualitative example on co-speech gesture generation. Given a speech, we visualize the ground truth 3D motion accompanying the audio, the motion generated by the baseline EMAGE liu24emage, SynTalker chen2024Synerg and our method. Our model generates more diverse and expressive motion compared to the baseline, especially when the speaker emphasizes on certain words such as "tired" and "because".
• Figure 5: Generation performance vs. the amount of post-training data. Our model learns a stronger motion prior from pre-training and thus shows much better under data scarcity.