Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMM: Generative Masked Motion Model

Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, Chen Chen

TL;DR

MMM addresses the speed, fidelity, and editability gap in text‑to‑motion generation by introducing a two‑stage framework: a Motion Tokenizer that maps 3D motion to discrete tokens via a large VQ‑VAE codebook, and a Conditional Masked Motion Model that reconstructs masked motion tokens from text and context, enabling parallel and iterative decoding. Training optimizes a vector‑quantization loss and a masked‑token reconstruction loss, while inference uses a decaying mask schedule to progressively unveil tokens, achieving real‑time generation and natural editing through mask placement. On HumanML3D t2m and KIT‑ML KIT, MMM delivers state‑of‑the‑art quality (low FID and strong multimodal alignment) with substantially faster inference (AITS around 0.081 s on a mid‑range GPU) and supports editing tasks such as in-betweening and upper‑body edits, as well as long‑range generation via transition tokens. The approach yields a practical, scalable solution for animation, VR/AR, and robotics, where high‑fidelity motion must be produced quickly and edited interactively.

Abstract

Recent advances in text-to-motion generation using diffusion and autoregressive models have shown promising results. However, these models often suffer from a trade-off between real-time performance, high fidelity, and motion editability. To address this gap, we introduce MMM, a novel yet simple motion generation paradigm based on Masked Motion Model. MMM consists of two key components: (1) a motion tokenizer that transforms 3D human motion into a sequence of discrete tokens in latent space, and (2) a conditional masked motion transformer that learns to predict randomly masked motion tokens, conditioned on the pre-computed text tokens. By attending to motion and text tokens in all directions, MMM explicitly captures inherent dependency among motion tokens and semantic mapping between motion and text tokens. During inference, this allows parallel and iterative decoding of multiple motion tokens that are highly consistent with fine-grained text descriptions, therefore simultaneously achieving high-fidelity and high-speed motion generation. In addition, MMM has innate motion editability. By simply placing mask tokens in the place that needs editing, MMM automatically fills the gaps while guaranteeing smooth transitions between editing and non-editing parts. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that MMM surpasses current leading methods in generating high-quality motion (evidenced by superior FID scores of 0.08 and 0.429), while offering advanced editing features such as body-part modification, motion in-betweening, and the synthesis of long motion sequences. In addition, MMM is two orders of magnitude faster on a single mid-range GPU than editable motion diffusion models. Our project page is available at \url{https://exitudio.github.io/MMM-page}.

MMM: Generative Masked Motion Model

TL;DR

MMM addresses the speed, fidelity, and editability gap in text‑to‑motion generation by introducing a two‑stage framework: a Motion Tokenizer that maps 3D motion to discrete tokens via a large VQ‑VAE codebook, and a Conditional Masked Motion Model that reconstructs masked motion tokens from text and context, enabling parallel and iterative decoding. Training optimizes a vector‑quantization loss and a masked‑token reconstruction loss, while inference uses a decaying mask schedule to progressively unveil tokens, achieving real‑time generation and natural editing through mask placement. On HumanML3D t2m and KIT‑ML KIT, MMM delivers state‑of‑the‑art quality (low FID and strong multimodal alignment) with substantially faster inference (AITS around 0.081 s on a mid‑range GPU) and supports editing tasks such as in-betweening and upper‑body edits, as well as long‑range generation via transition tokens. The approach yields a practical, scalable solution for animation, VR/AR, and robotics, where high‑fidelity motion must be produced quickly and edited interactively.

Abstract

Recent advances in text-to-motion generation using diffusion and autoregressive models have shown promising results. However, these models often suffer from a trade-off between real-time performance, high fidelity, and motion editability. To address this gap, we introduce MMM, a novel yet simple motion generation paradigm based on Masked Motion Model. MMM consists of two key components: (1) a motion tokenizer that transforms 3D human motion into a sequence of discrete tokens in latent space, and (2) a conditional masked motion transformer that learns to predict randomly masked motion tokens, conditioned on the pre-computed text tokens. By attending to motion and text tokens in all directions, MMM explicitly captures inherent dependency among motion tokens and semantic mapping between motion and text tokens. During inference, this allows parallel and iterative decoding of multiple motion tokens that are highly consistent with fine-grained text descriptions, therefore simultaneously achieving high-fidelity and high-speed motion generation. In addition, MMM has innate motion editability. By simply placing mask tokens in the place that needs editing, MMM automatically fills the gaps while guaranteeing smooth transitions between editing and non-editing parts. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that MMM surpasses current leading methods in generating high-quality motion (evidenced by superior FID scores of 0.08 and 0.429), while offering advanced editing features such as body-part modification, motion in-betweening, and the synthesis of long motion sequences. In addition, MMM is two orders of magnitude faster on a single mid-range GPU than editable motion diffusion models. Our project page is available at \url{https://exitudio.github.io/MMM-page}.
Paper Structure (24 sections, 5 equations, 14 figures, 13 tables)

This paper contains 24 sections, 5 equations, 14 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Motion Tokenizer
  5. Conditional Masked Motion Model
  6. Motion Editing
  7. Experiments
  8. Comparison to State-of-the-art Approaches
  9. Inference Speed and Editability
  10. Ablation Study
  11. Conclusion
  12. Overview
  13. Inference speed, quality, and editability
  14. Codebook Reset
  15. Influence of Word Embedding via Cross Attention
  16. ...and 9 more sections

Figures (14)

  • Figure 1: The motion generation quality (FID score) and speed (AITS) comparisons between MMM and SOTA methods on HumanML3D dataset. The model closer to the origin is better. MMM achieves the best FID score (0.08) and the highest speed (0.081 AITS), while preserving motion editability. "$\bigcirc$" represents editibility and "$\times$" otherwise. All tests are performed on a single NVIDIA RTX A5000.
  • Figure 2: Overall architecture of MMM. (a) Motion Tokenizer transforms the raw motion sequence into discrete motion tokens according to a learned codebook. (b) Conditional Masked Transformer learns to predict masked motion tokens, conditioned on word and sentence tokens obtained from CLIP text encoders. (c) Motion Generation starts from an empty canvas and the masked transformer concurrently and progressively predicts multiple high-confidence motion tokens.
  • Figure 3: Motion Editing. (Left) Motion in-betweening. (Middle) Long Sequence Generation. (Right) Upper Body Editing. "M" refers to mask token. "T" is text conditioned tokens and "L" denotes lower body part conditioned tokens.
  • Figure 4: Qualitative comparison of state-of-the-art methods with textual description: "a person walks forward then turns completely around and does a cartwheel." MDM is one of the most representative motion-space diffusion models. MLD is the first and SOTA latent-space motion diffusion model. T2M-GPT is the first and SOTA autoregressive motion model. Top-left: MDM MDM does not execute cartwheel motion. Top-right: MLD MLD generates unrealistic motion and lacks a complete cartwheel motion. Middle-left: the trajectory of T2M-GPT T2M-GPT is not "completely around". Middle-right: our method generates realistic motion and trajectory compared to the ground truth on the bottom. Trajectories start from blue and end in red.
  • Figure 5: Qualitative comparison of upper body editing, generating upper body part based on the text "a man throws a ball" conditioned on lower body part of "a man rises from the ground, walks in a circle and sits back down on the ground."
  • ...and 9 more figures