Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions

Wendong Bu, Kaihang Pan, Yuze Lin, Jiacheng Li, Kai Shen, Wenqiao Zhang, Juncheng Li, Jun Xiao, Siliang Tang

TL;DR

OmniMoGen introduces a unified framework for human motion generation driven by interleaved text-motion instructions, addressing the fragmentation of tasks like text-to-motion and editing. It pairs an RVQ-VAE motion tokenizer with a transformer, trained on the X2Mo dataset and evaluated on AnyContext. The approach achieves state-of-the-art results across multiple tasks and reveals emergent capabilities such as compositional editing, self-reflection, and knowledge-informed generation. This work advances toward intuitive, free-form motion generation and provides a scalable benchmark suite for future research.

Abstract

Large language models (LLMs) have unified diverse linguistic tasks within a single framework, yet such unification remains unexplored in human motion generation. Existing methods are confined to isolated tasks, limiting flexibility for free-form and omni-objective generation. To address this, we propose OmniMoGen, a unified framework that enables versatile motion generation through interleaved text-motion instructions. Built upon a concise RVQ-VAE and transformer architecture, OmniMoGen supports end-to-end instruction-driven motion generation. We construct X2Mo, a large-scale dataset of over 137K interleaved text-motion instructions, and introduce AnyContext, a benchmark for evaluating interleaved motion generation. Experiments show that OmniMoGen achieves state-of-the-art performance on text-to-motion, motion editing, and AnyContext, exhibiting emerging capabilities such as compositional editing, self-reflective generation, and knowledge-informed generation. These results mark a step toward the next intelligent motion generation. Project Page: https://OmniMoGen.github.io/.

OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions

TL;DR

OmniMoGen introduces a unified framework for human motion generation driven by interleaved text-motion instructions, addressing the fragmentation of tasks like text-to-motion and editing. It pairs an RVQ-VAE motion tokenizer with a transformer, trained on the X2Mo dataset and evaluated on AnyContext. The approach achieves state-of-the-art results across multiple tasks and reveals emergent capabilities such as compositional editing, self-reflection, and knowledge-informed generation. This work advances toward intuitive, free-form motion generation and provides a scalable benchmark suite for future research.

Abstract

Large language models (LLMs) have unified diverse linguistic tasks within a single framework, yet such unification remains unexplored in human motion generation. Existing methods are confined to isolated tasks, limiting flexibility for free-form and omni-objective generation. To address this, we propose OmniMoGen, a unified framework that enables versatile motion generation through interleaved text-motion instructions. Built upon a concise RVQ-VAE and transformer architecture, OmniMoGen supports end-to-end instruction-driven motion generation. We construct X2Mo, a large-scale dataset of over 137K interleaved text-motion instructions, and introduce AnyContext, a benchmark for evaluating interleaved motion generation. Experiments show that OmniMoGen achieves state-of-the-art performance on text-to-motion, motion editing, and AnyContext, exhibiting emerging capabilities such as compositional editing, self-reflective generation, and knowledge-informed generation. These results mark a step toward the next intelligent motion generation. Project Page: https://OmniMoGen.github.io/.
Paper Structure (44 sections, 25 equations, 8 figures, 8 tables)

This paper contains 44 sections, 25 equations, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. X2Mo Dataset
  5. Interleaved Text-Motion Instructions
  6. Data Construction
  7. OmniMoGen
  8. Model Architecture.
  9. Supervised Fine-Tuning.
  10. Reinforcement Learning.
  11. AnyContext Benchmark
  12. Task Categorization
  13. Evaluation Metrics
  14. Experiments
  15. Experimental Setup
  16. ...and 29 more sections

Figures (8)

  • Figure 1: Similar to ChatGPT in NLP, OmniMoGen unifies all motion generation tasks in a unified architecture, such as text-to-motion, style editing, trajectory editing, inpainting, in-betweening, compositional editing, self-reflective generation, and knowledge-informed generation. OmniMoGen enables seamless and flexible motion generation across diverse objectives by merely adjusting the interleaved text-motion instructions.
  • Figure 2: An overview of OmniMoGen, comprising (a) an RVQ-VAE and (b) an autoregressive transformer. Motions are encoded into discrete tokens like a foreign language by the RVQ-VAE, and then concatenated with text tokens as input to a unified autoregressive transformer for next-token prediction.
  • Figure 3: Task types and interleaved text–motion instruction formats in AnyContext
  • Figure 4: Qualitative comparison of text-to-motion generation on HumanML3D. The red words and boxes highlight the misaligned motions.
  • Figure 5: Qualitative comparison of motion editing on MotionFix. The red words and boxes highlight the misaligned motions.
  • ...and 3 more figures