OmniMotion: Multimodal Motion Generation with Continuous Masked Autoregression

TL;DR

This work tackles multimodal whole-body motion generation by unifying text, speech, and music conditioning within a single framework. It introduces a continuous masked autoregressive motion transformer with causal attention, enhanced by a gated linear mechanism and RMSNorm, followed by diffusion-based refinement via a Diffusion Transformer (DiT). Multimodal fusion is achieved through AdaLN and cross-attention, with text-to-motion pretraining and multimodal control adaptation enabling cross-domain generalization. Empirical results on HumanML3D, BEAT2, and FineDance demonstrate superior performance across text-to-motion, speech-to-gesture, and music-to-dance tasks, showing the practical potential of a unified multimodal motion generation pipeline. Key equations include the masking schedule $\gamma(\tau) = \cos(\frac{\pi \tau}{2})$, the diffusion-based sampling update, and the CFG-guided conditioning, all of which are wrapped in $...$ to emphasize their mathematical form.

Abstract

Whole-body multi-modal human motion generation poses two primary challenges: creating an effective motion generation mechanism and integrating various modalities, such as text, speech, and music, into a cohesive framework. Unlike previous methods that usually employ discrete masked modeling or autoregressive modeling, we develop a continuous masked autoregressive motion transformer, where a causal attention is performed considering the sequential nature within the human motion. Within this transformer, we introduce a gated linear attention and an RMSNorm module, which drive the transformer to pay attention to the key actions and suppress the instability caused by either the abnormal movements or the heterogeneous distributions within multi-modalities. To further enhance both the motion generation and the multimodal generalization, we employ the DiT structure to diffuse the conditions from the transformer towards the targets. To fuse different modalities, AdaLN and cross-attention are leveraged to inject the text, speech, and music signals. Experimental results demonstrate that our framework outperforms previous methods across all modalities, including text-to-motion, speech-to-gesture, and music-to-dance. The code of our method will be made public.

Figures (6)

• Figure 1: We construct an omni motion framework with a continuous masked autoregressive motion transformer for multimodal whole-body motion modeling, including text-based, music-based, and speech-based motion generation.
• Figure 2: Framework overview. Our framework consists of three parts: (a) The input motion is encoded by an autoencoder to extract a latent code, producing the continuous motion tokens. (b) The motion tokens are masked and predicted in an autoregressive transformer with causal attention, producing conditions for DiTs to diffuse towards the target tokens. (c) Multimodal signals are encoded and then injected via AdaLN and cross-attention.
• Figure 3: The qualitative results of motions generated from our model driven by speech and music.
• Figure 4: The qualitative results of text-driven motion generation.
• Figure 5: The qualitative results of speech-driven motion generation.