SegMo: Segment-aligned Text to 3D Human Motion Generation

TL;DR

SegMo addresses the coarse nature of sequence-level text–motion alignment by introducing segment-aligned generation for 3D human motion from text. It decomposes both text and motion into temporally ordered segments via Text Segment Extraction (LLMs) and Motion Segment Extraction (uniform/CPD/clustering), and trains a segment-wise contrastive alignment in a shared embedding space atop a Residual VQ-VAE backbone with a Mask Transformer and a Residual Transformer. The approach yields improved text–motion alignment and realism on HumanML3D and KIT-ML, and enables retrieval tasks such as motion grounding and motion-to-text retrieval without language-motion pretraining. Ablation studies validate the benefits of segment-level design and the stability of uniform segmentation, while also highlighting segmentation precision as a future improvement area with potential gains in fine-grained control. Overall, SegMo advances fine-grained controllability and applicability of text-conditioned motion generation in graphics, VR/AR, and related fields.

Abstract

Generating 3D human motions from textual descriptions is an important research problem with broad applications in video games, virtual reality, and augmented reality. Recent methods align the textual description with human motion at the sequence level, neglecting the internal semantic structure of modalities. However, both motion descriptions and motion sequences can be naturally decomposed into smaller and semantically coherent segments, which can serve as atomic alignment units to achieve finer-grained correspondence. Motivated by this, we propose SegMo, a novel Segment-aligned text-conditioned human Motion generation framework to achieve fine-grained text-motion alignment. Our framework consists of three modules: (1) Text Segment Extraction, which decomposes complex textual descriptions into temporally ordered phrases, each representing a simple atomic action; (2) Motion Segment Extraction, which partitions complete motion sequences into corresponding motion segments; and (3) Fine-grained Text-Motion Alignment, which aligns text and motion segments with contrastive learning. Extensive experiments demonstrate that SegMo improves the strong baseline on two widely used datasets, achieving an improved TOP 1 score of 0.553 on the HumanML3D test set. Moreover, thanks to the learned shared embedding space for text and motion segments, SegMo can also be applied to retrieval-style tasks such as motion grounding and motion-to-text retrieval.

Figures (9)

• Figure 1: The main idea of our method. We decompose the complex motion description and motion sequence into simpler temporally ordered segments and align them in a shared embedding space to improve the accuracy and realism of generated motions.
• Figure 2: The overview of our method. Left: The Residual VQ-VAE encodes a continuous motion sequence into discrete motion tokens, including base tokens and residual tokens, which will be generated by the mask transformer and residual transformer, respectively. Right: The Mask Transformer predicts the masked base tokens conditioned on the textual description. To achieve segment-level fine-grained alignment, we introduce a Text Segment Extraction module and a Motion Segment Extraction module, which extract text and motion segments respectively, and align them through the Fine-grained Text-Motion Alignment module.
• Figure 3: Qualitative comparison of T2M-GPT, MoMask, and Ours on the HumanML3D test set. A fixed number of keyframes is shown for each motion sequence. Please refer to the supplementary video for additional comparison results.
• Figure 4: The Intra-Segment Consistency (ISC) of all models evaluated under different segmentation methods on the HumanML3D test set. The Coefficient of Variation (CV), defined as $\mathrm{std}(ISC)/\mathrm{mean}(ISC)$, is reported to assess stability. "Train" denotes the segmentation method used for training, while "Eval" denotes the segmentation method used for evaluation.
• Figure 5: Example of motion grounding. Top: Results using the similarity map generated by our method. Bottom: Similarity map generated by MoMask MoMask. In each map, the x-axis denotes the start index of the sliding window and the y-axis denotes the text segment. The motion length is 49, and the window size is 5.