UniMo: Unified Motion Generation and Understanding with Chain of Thought

TL;DR

UniMo presents a unified, interpretable framework for 3D human motion generation and understanding by integrating motion-grounded chain-of-thought reasoning into an LLM, followed by GRPO-based reinforcement learning with task-specific rewards. It introduces a VQ-VAE motion tokenizer and a two-stage training pipeline (SFT with CoT, then GRPO RL) to tightly couple text- and motion-based tasks. Experiments on HumanML3D with curated CoT annotations show state-of-the-art performance on both Text-to-Motion and Motion-to-Text, with ablations validating CoT usefulness, RL rewards, and unified modeling. The work advances bidirectional grounding between language and motion and enables interpretable, stepwise reasoning for controllable animation and multimodal understanding.

Abstract

Existing 3D human motion generation and understanding methods often exhibit limited interpretability, restricting effective mutual enhancement between these inherently related tasks. While current unified frameworks based on large language models (LLMs) leverage linguistic priors, they frequently encounter challenges in semantic alignment and task coherence. Moreover, the next-token prediction paradigm in LLMs is ill-suited for motion sequences, causing cumulative prediction errors. To address these limitations, we propose UniMo, a novel framework that integrates motion-language information and interpretable chain of thought (CoT) reasoning into the LLM via supervised fine-tuning (SFT). We further introduce reinforcement learning with Group Relative Policy Optimization (GRPO) as a post-training strategy that optimizes over groups of tokens to enforce structural correctness and semantic alignment, mitigating cumulative errors in motion token prediction. Extensive experiments demonstrate that UniMo significantly outperforms existing unified and task-specific models, achieving state-of-the-art performance in both motion generation and understanding.

Figures (9)

• Figure 1: The UniMo is trained in two stages: the SFT stage and the reinforcement learning stage with GRPO. In the SFT stage, the model is teached to perform both T2M and M2T tasks with structured reasoning, i.e. CoT. In the RL stage, the model is further optimized with task-specific rewards, enabling unified and interpretable motion generation and understanding.
• Figure 2: Illustration of the CoT annotation process. Human joint sequences are rendered in the Blender and paired with captions, which are further processed by the Qwen2.5-VL-72B to generate reasoning traces.
• Figure 3: Qualitative comparison of our method with other open-source SOTAs such as MoMask Momask and MotionLLM MotionLLM for the text-to-motion task. Our method presents stronger instruction-following capability and can generate sequential actions. We highly recommend the readers to watch the video comparisons in supplementary materials.
• Figure 4: Qualitative comparison of our method with other open-source SOTAs such as MotionGPT MotionGPT and MotionLLM MotionLLM for the motion-to-text task. Our method can more precisely describe complex motions.
• Figure 5: Comparative word-clouds highlighting the most frequent textual cues in HumanML3D captions on the left and the corresponding CoT annotations on the right.