MotionGPT: Human Motion as a Foreign Language

TL;DR

MotionGPT presents a unified framework that treats human motion as a language by discretizing motions into tokens with a VQ-VAE and training a shared language model on a combined motion-language vocabulary. A three-stage training pipeline—motion tokenizer pre-training, motion-language pre-training, and instruction tuning—enables a single model to perform text-to-motion, motion-to-text, motion prediction, and motion in-between with prompts. The approach achieves competitive or state-of-the-art results across multiple motion tasks on HumanML3D and KIT, with extensive ablations and analyses demonstrating the impact of codebook size, model scale, and training strategy. The work highlights the potential of large-language-model-based frameworks for multi-task motion understanding and generation, offering a scalable, user-friendly paradigm for motion-aware AI applications in gaming, robotics, and behavior analysis.

Abstract

Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

Figures (9)

• Figure 1: MotionGPT can address diverse motion-relevant tasks uniformly given different instructions. We provide the results on text-to-motion (the upper left), motion captioning (the bottom left), motion completion (the upper right), and the language question-to-answer (the bottom right). The left to right of motion represents the time order. Blue motion denotes the input, and yellow is the generation.
• Figure 2: Method overview: MotionGPT consists of a motion tokenizer $\mathcal{V}$ (\ref{['sec:method:vqvae']}) and a motion-aware language model (\ref{['sec:method:lm']}). Combining Motion Tokens learned by $\mathcal{V}$ and Text Tokens by text tokenizer, we then learn motion and language jointly utilizing language model as backbone.
• Figure 3: Training Scheme. We introduce three training steps for our MotionGPT (\ref{['sec:method: strategy']}): First $\mathcal{V}$ learn a codebook for discrete motion representation. Then we train language using a mixture of language and motion data to learn the semantic coupling between text and motion. Finally, we fine-tune the model in a multi-task text-motion dataset with instructions.
• Figure 4: Comparison on text-driven motion generation. The provided state-of-the-art methods are under the same training and inference setting on HumanML3D Guo_2022_CVPR_humanml3d. The red words and boxes highlight the misaligned motions. The results demonstrate that our motion-language per-training shows promising text understanding for motion generation.
• Figure 5: Gallery for the results of our unified MotionGPT. More samples are from our best model for text-to-motion synthesis, motion captioning, and textual question-to-answer task. The supervision of MotionGPT relies on our instruction-based motion-language dataset ($cf.$\ref{['sec:appendix:data: details']}) based on previous motion datasets Guo_2022_CVPR_humanml3dAMASS_ICCV2019. We recommend the dynamic visualization in our supplemental video.