Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiMo: Discrete Diffusion Modeling for Motion Generation and Understanding

Ning Zhang, Zhengyu Li, Kwong Weng Loh, Mingxi Xu, Qi Wang, Zhengyu Wen, Xiaoyu He, Wei Zhao, Kehong Gong, Mingyuan Zhang

TL;DR

DiMo introduces a unified discrete diffusion framework for bidirectional text–motion modeling, treating text and motion as noisy sequences and performing parallel, multi-step denoising to generate and interpret motion from text and vice versa. By integrating Residual Vector Quantization for high-fidelity motion tokens, a BERT-based language backbone, multi-task scheduling, and confidence-guided progressive inference, DiMo enables a tunable quality–latency trade-off and supports text-free motion completion and caption correction without architectural changes. GRPO fine-tuning further enhances cross-modal alignment, yielding strong perceptual motion quality and robust text generation on HumanML3D and KIT-ML, with competitive T2M and M2T performance across metrics and notable qualitative improvements through iterative refinement. The approach demonstrates the practicality of a single diffusion-based model for unified motion-language understanding and generation, paving the way for scalable, editable, and controllable cross-modal motion systems.

Abstract

Prior masked modeling motion generation methods predominantly study text-to-motion. We present DiMo, a discrete diffusion-style framework, which extends masked modeling to bidirectional text--motion understanding and generation. Unlike GPT-style autoregressive approaches that tokenize motion and decode sequentially, DiMo performs iterative masked token refinement, unifying Text-to-Motion (T2M), Motion-to-Text (M2T), and text-free Motion-to-Motion (M2M) within a single model. This decoding paradigm naturally enables a quality-latency trade-off at inference via the number of refinement steps.We further improve motion token fidelity with residual vector quantization (RVQ) and enhance alignment and controllability with Group Relative Policy Optimization (GRPO). Experiments on HumanML3D and KIT-ML show strong motion quality and competitive bidirectional understanding under a unified framework. In addition, we demonstrate model ability in text-free motion completion, text-guided motion prediction and motion caption correction without architectural change.Additional qualitative results are available on our project page: https://animotionlab.github.io/DiMo/.

DiMo: Discrete Diffusion Modeling for Motion Generation and Understanding

TL;DR

DiMo introduces a unified discrete diffusion framework for bidirectional text–motion modeling, treating text and motion as noisy sequences and performing parallel, multi-step denoising to generate and interpret motion from text and vice versa. By integrating Residual Vector Quantization for high-fidelity motion tokens, a BERT-based language backbone, multi-task scheduling, and confidence-guided progressive inference, DiMo enables a tunable quality–latency trade-off and supports text-free motion completion and caption correction without architectural changes. GRPO fine-tuning further enhances cross-modal alignment, yielding strong perceptual motion quality and robust text generation on HumanML3D and KIT-ML, with competitive T2M and M2T performance across metrics and notable qualitative improvements through iterative refinement. The approach demonstrates the practicality of a single diffusion-based model for unified motion-language understanding and generation, paving the way for scalable, editable, and controllable cross-modal motion systems.

Abstract

Prior masked modeling motion generation methods predominantly study text-to-motion. We present DiMo, a discrete diffusion-style framework, which extends masked modeling to bidirectional text--motion understanding and generation. Unlike GPT-style autoregressive approaches that tokenize motion and decode sequentially, DiMo performs iterative masked token refinement, unifying Text-to-Motion (T2M), Motion-to-Text (M2T), and text-free Motion-to-Motion (M2M) within a single model. This decoding paradigm naturally enables a quality-latency trade-off at inference via the number of refinement steps.We further improve motion token fidelity with residual vector quantization (RVQ) and enhance alignment and controllability with Group Relative Policy Optimization (GRPO). Experiments on HumanML3D and KIT-ML show strong motion quality and competitive bidirectional understanding under a unified framework. In addition, we demonstrate model ability in text-free motion completion, text-guided motion prediction and motion caption correction without architectural change.Additional qualitative results are available on our project page: https://animotionlab.github.io/DiMo/.
Paper Structure (43 sections, 17 equations, 5 figures, 15 tables, 1 algorithm)

This paper contains 43 sections, 17 equations, 5 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Human Motion Generation
  4. Bi-directional Motion-Text Generation
  5. Diffusion Language Models (dLLMs)
  6. Methods
  7. Overview
  8. DiMo
  9. Multi-task Scheduling.
  10. Masking Schedule and Training Loss
  11. Confidence-Guided Progressive Inference.
  12. Motion Tokenization with Residual VQ
  13. Model Architecture
  14. Language backbone.
  15. Motion token encoder.
  16. ...and 28 more sections

Figures (5)

  • Figure 1: Overview of DiMo. DiMo unifies Motion-to-Text (M2T) and Text-to-Motion (T2M) within a single framework, achieving a strong balance between motion realism and semantic consistency across generation and understanding tasks.
  • Figure 2: Overview of DiMo. Our unified framework supports text-to-motion (T2M), motion-to-text (M2T), and motion-to-motion (M2M) tasks with RVQ-based motion tokenization, multi-task masked training, confidence-guided progressive inference, and GRPO fine-tuning.
  • Figure 3: Text-to-motion comparison: DiMo generates more coherent and semantically aligned motions.
  • Figure 4: Motion-to-text comparison: DiMo produces concise and accurate action descriptions.
  • Figure 5: Application: Examples of downstream tasks enabled by DiMo