MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators

TL;DR

MotionGPT introduces a unified framework for multimodal human motion generation by fine-tuning an LLM with motion instructions that combine text and pose controls. Motion codes are produced via a motion VQ-VAE and then decoded to motion sequences, enabling the model to handle diverse tasks (text-to-motion, pose-conditioned generation) within a single architecture using LoRA-based instruction tuning. Experiments on HumanML3D and KIT-ML show MotionGPT achieves competitive motion quality under multiple control conditions, with improvements when using pre-trained LLMs and when training jointly across tasks. The approach demonstrates a practical, flexible direction for programmable digital humans, with potential extensions to additional modalities beyond text and poses.

Abstract

Generating realistic human motion from given action descriptions has experienced significant advancements because of the emerging requirement of digital humans. While recent works have achieved impressive results in generating motion directly from textual action descriptions, they often support only a single modality of the control signal, which limits their application in the real digital human industry. This paper presents a Motion General-Purpose generaTor (MotionGPT) that can use multimodal control signals, e.g., text and single-frame poses, for generating consecutive human motions by treating multimodal signals as special input tokens in large language models (LLMs). Specifically, we first quantize multimodal control signals into discrete codes and then formulate them in a unified prompt instruction to ask the LLMs to generate the motion answer. Our MotionGPT demonstrates a unified human motion generation model with multimodal control signals by tuning a mere 0.4% of LLM parameters. To the best of our knowledge, MotionGPT is the first method to generate human motion by multimodal control signals, which we hope can shed light on this new direction. Visit our webpage at https://qiqiapink.github.io/MotionGPT/.

Figures (8)

• Figure 1: This work proposes a novel human motion generation method via fine-tuned LLMs, named MotionGPT. Compared with previous methods, MotionGPT has the unique ability to accept multiple control conditions and solve various motion generation tasks using a unified model.
• Figure 2: The pipeline of MotionGPT, a Motion General-Purpose generaTor. Given text and poses as an input example, we organize task descriptions (Instruction) and multiple control conditions (Input) within a question template. MotionGPT fine-tunes an LLM to generate the corresponding motion answer, which can then be decoded into human motions using a VQ-VAE decoder.
• Figure 3: Generated motion by MotionGPT with multiple control conditions on HumanML3D.
• Figure 4: Qualitative comparison of the state-of-the-art motion generation method MDM with text-only conditions on HumanML3D.
• Figure 5: More text-to-motion samples generated by MotionGPT-13B using texts from the HumanML3D test set.