Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mogo: RQ Hierarchical Causal Transformer for High-Quality 3D Human Motion Generation

Dongjie Fu

TL;DR

Mogo introduces a GPT-type architecture that delivers high-quality text-to-3D motion generation with streaming output and strong generalization. It fuses RVQ-VAE-based motion tokenization with a single Hierarchical Causal Transformer to jointly generate base motions and layer-wise residuals, enabling continuous sequences up to 13 seconds and real-time inference. Across HumanML3D, KIT-ML, and CMP, Mogo achieves state-of-the-art results among GPT-type models and competitive performance against BERT-type approaches, including zero-shot CMP evaluation and a user-preference advantage. The approach demonstrates practical impact for open-vocabulary prompts and interactive media applications, supported by prompt engineering and rigorous ablations on codebook design and transformer architecture.

Abstract

In the field of text-to-motion generation, Bert-type Masked Models (MoMask, MMM) currently produce higher-quality outputs compared to GPT-type autoregressive models (T2M-GPT). However, these Bert-type models often lack the streaming output capability required for applications in video game and multimedia environments, a feature inherent to GPT-type models. Additionally, they demonstrate weaker performance in out-of-distribution generation. To surpass the quality of BERT-type models while leveraging a GPT-type structure, without adding extra refinement models that complicate scaling data, we propose a novel architecture, Mogo (Motion Only Generate Once), which generates high-quality lifelike 3D human motions by training a single transformer model. Mogo consists of only two main components: 1) RVQ-VAE, a hierarchical residual vector quantization variational autoencoder, which discretizes continuous motion sequences with high precision; 2) Hierarchical Causal Transformer, responsible for generating the base motion sequences in an autoregressive manner while simultaneously inferring residuals across different layers. Experimental results demonstrate that Mogo can generate continuous and cyclic motion sequences up to 260 frames (13 seconds), surpassing the 196 frames (10 seconds) length limitation of existing datasets like HumanML3D. On the HumanML3D test set, Mogo achieves a FID score of 0.079, outperforming both the GPT-type model T2M-GPT (FID = 0.116), AttT2M (FID = 0.112) and the BERT-type model MMM (FID = 0.080). Furthermore, our model achieves the best quantitative performance in out-of-distribution generation.

Mogo: RQ Hierarchical Causal Transformer for High-Quality 3D Human Motion Generation

TL;DR

Mogo introduces a GPT-type architecture that delivers high-quality text-to-3D motion generation with streaming output and strong generalization. It fuses RVQ-VAE-based motion tokenization with a single Hierarchical Causal Transformer to jointly generate base motions and layer-wise residuals, enabling continuous sequences up to 13 seconds and real-time inference. Across HumanML3D, KIT-ML, and CMP, Mogo achieves state-of-the-art results among GPT-type models and competitive performance against BERT-type approaches, including zero-shot CMP evaluation and a user-preference advantage. The approach demonstrates practical impact for open-vocabulary prompts and interactive media applications, supported by prompt engineering and rigorous ablations on codebook design and transformer architecture.

Abstract

In the field of text-to-motion generation, Bert-type Masked Models (MoMask, MMM) currently produce higher-quality outputs compared to GPT-type autoregressive models (T2M-GPT). However, these Bert-type models often lack the streaming output capability required for applications in video game and multimedia environments, a feature inherent to GPT-type models. Additionally, they demonstrate weaker performance in out-of-distribution generation. To surpass the quality of BERT-type models while leveraging a GPT-type structure, without adding extra refinement models that complicate scaling data, we propose a novel architecture, Mogo (Motion Only Generate Once), which generates high-quality lifelike 3D human motions by training a single transformer model. Mogo consists of only two main components: 1) RVQ-VAE, a hierarchical residual vector quantization variational autoencoder, which discretizes continuous motion sequences with high precision; 2) Hierarchical Causal Transformer, responsible for generating the base motion sequences in an autoregressive manner while simultaneously inferring residuals across different layers. Experimental results demonstrate that Mogo can generate continuous and cyclic motion sequences up to 260 frames (13 seconds), surpassing the 196 frames (10 seconds) length limitation of existing datasets like HumanML3D. On the HumanML3D test set, Mogo achieves a FID score of 0.079, outperforming both the GPT-type model T2M-GPT (FID = 0.116), AttT2M (FID = 0.112) and the BERT-type model MMM (FID = 0.080). Furthermore, our model achieves the best quantitative performance in out-of-distribution generation.

Paper Structure

This paper contains 23 sections, 6 equations, 9 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Training: Motion Residual VQ-VAE
  5. Training: Hierarchical Causal Transformer
  6. Inference Capability Optimization
  7. Experiments
  8. Datasets and Evaluation Metrics
  9. Implementation Details
  10. Inference
  11. Comparison to state-of-the-art approaches
  12. User Study
  13. Ablation Study
  14. Limitations and Discussion
  15. Conclusion
  16. ...and 8 more sections

Figures (9)

  • Figure 1: Mogo is a novel GPT-type model for high-quality text-to-motion generation, capable of producing longer sequences and handling open-vocabulary prompts. Given inputs e.g."A person is dancing ballet gracefully", "A person walks like a zombie". Mogo generates vivid, lifelike 3D human motions at once.
  • Figure 2: The architecture of the model:(a) RVQ-VAE is a hierarchical residual vector quantization variational autoencoder that discretizes continuous motion sequences with high precision; (b) a Rq Hierarchical Causal Transformer generates base motion sequences autoregressively while inferring residuals across layers.
  • Figure 3: The architecture of prompt engineering for enhanced model inference
  • Figure 4: Comparison of the generation quality between our model and the latest SOTA motion generation modelsguo2023momaskjiang2024motiongpt.
  • Figure 5: HumanML3D User Study
  • ...and 4 more figures