MoGenTS: Motion Generation based on Spatial-Temporal Joint Modeling

TL;DR

MoGenTS tackles text-driven human motion generation by quantizing each joint into individual codes to form a spatial-temporal 2D token map. A temporal-spatial 2D masking strategy and a spatial-temporal 2D transformer conditioned on text enable robust generation that preserves both joint-level spatial relationships and temporal motion patterns. The approach yields significant improvements over state-of-the-art methods on HumanML3D and KIT-ML, reducing FID and improving motion fidelity, while also enabling motion editing. Overall, the work demonstrates that joint-wise quantization plus 2D token modeling and attention can effectively leverage 2D operations for high-quality, text-guided motion synthesis.

Abstract

Motion generation from discrete quantization offers many advantages over continuous regression, but at the cost of inevitable approximation errors. Previous methods usually quantize the entire body pose into one code, which not only faces the difficulty in encoding all joints within one vector but also loses the spatial relationship between different joints. Differently, in this work we quantize each individual joint into one vector, which i) simplifies the quantization process as the complexity associated with a single joint is markedly lower than that of the entire pose; ii) maintains a spatial-temporal structure that preserves both the spatial relationships among joints and the temporal movement patterns; iii) yields a 2D token map, which enables the application of various 2D operations widely used in 2D images. Grounded in the 2D motion quantization, we build a spatial-temporal modeling framework, where 2D joint VQVAE, temporal-spatial 2D masking technique, and spatial-temporal 2D attention are proposed to take advantage of spatial-temporal signals among the 2D tokens. Extensive experiments demonstrate that our method significantly outperforms previous methods across different datasets, with a 26.6% decrease of FID on HumanML3D and a 29.9% decrease on KIT-ML. Project page: https://aigc3d.github.io/mogents.

Figures (7)

• Figure 1: Framework overview. (a) In motion quantization, human motion is quantized into a spatial-temporal 2D token map by a joint VQ-VAE. (b) In motion generation, a temporal-spatial 2D masking is performed to obtain a masked map, and then a spatial-temporal 2D transformer is designed to infer the masked tokens.
• Figure 2: The structure of our spatial-temporal 2D Joint VQ-VAE for motion quantization.
• Figure 3: The temporal-spatial masking strategy (a) and the spatial-temporal attention (b) for motion generation.
• Figure 4: Qualitative results on the test set of HumanML3D. The color from light blue to dark blue indicates the motion sequence order. An arrow indicates this sequence is unfolded in the time axis.
• Figure 5: Motion Editing. The edited regions are indicated in green.